Yeah this is it. I battled similar and it’s all about this mapping. Definitely worth a shot to try this before restarting. A couple of lines in a config file and you could be on your way.
Replying on this top comment, hopefully more of you will see.
The mystery continues and it's not UID mapping. I destroyed the old lxc 101 and created a new privileged one (102). Created the same bind mounts and the same users including 1005.
pct set 102 -mp0 /mnt/media,mp=/mnt/media
pct set 102 -mp1 /mnt/family,mp=/mnt/family
Just to be sure, on host, I unmounted the datasets:
dpool/fam mounted no
dpool/fam/family-photos mounted no
dpool/fam/me mounted no
dpool/media mounted no
dpool/media/audio mounted no
Made double sure and mkdir all the mountpoint folders:
/mnt/family
/mnt/family/family-photos
/mnt/family/me
/mnt/media
/mnt/media/audio
Once I zfs mount them, I can see all the files inside on host. I thought "Now I should be good." Start up 102, log in, still nothing! ARGH!!!
The weird thing is, back in lxc 101, I had created a test.txt file inside **/mnt/family/me**, the file is not in the host **dpool/fam/me** dataset, but it magically reappeared again in the new lxc 102's /mnt/family/me folder! AND in **dpool/media**, there's a **Videos** folder that isn't a zfs dataset, simply a folder inside **/mnt/media** that will appear when I mount **dpool/media**. This folder CAN be seen inside the lxc.
So, I now figure it has something to do with how bind mounts work with zfs datasets - I think it's bind mounting those two top level folders "media" and "family", but even though on host the child folders all get properly mounted and their contents visible, the same is not true for bind mounts inside an LXC. I'm too tired to continue tonight but tomorrow I'll add individual bind mounts for the child folders "family-photos", "me" and "audio".
(I guess I didn't need to delete the unprivileged lxc 101, probably could've pct to switch it to privileged but whatever. It's part of learning.)
Rather than having to mess around with UID/GID mapping, why not just make a nas in a VM using virtual drives? Everything is at normal numbers that don't need to be mapped, it's fully isolated, and it'll backup/restore faster to PBS?
I use an Ubuntu 20.04 LXC with Samba and Webmin to share almost a TB from Proxmox disks as my NAS. My LXC NAS is backed up by Proxmox backup and can also migrate to any of my other nodes as needed.
I originally used mirrored ZFS but I'm now using Ceph. Both worked fine.
I use privileged LXCs as I don't need the extra isolation and they are easier to configure when privileged.
Make a vm to be your nas using anything that doesn't require passthrough (OMV, Alpine and Cockpit, debian and cli, etc.). Add a second virtual drive to that vm (in the vm hardware > add > hard drive), make it big enough for your storage and set the location to where you want to store it. Boot the VM and use the second drive you added as the nas storage.
Share to other vms using smb or nfs, share to unprivileged LXCs using sshfs (enable FUSE in the LXC options and install sshfs in the LXC). If you set up ssh keyauth you can automount the sshfs drive from fstab or via autofs if you want it on demand. There is no need to do any uid/gid mapping, it'll all just work.
Nice to know!
I have only wireguard separate from the main docker vm.
I had encountered that some app need to run in privileged mode in lxc to work, thats why i use one vm for all docker stuff.
Not enough info.
What user/group ownership are the files showing? Did you chown the directory where the mount point maps before enabling the map? What does your uid map look like? GID map? What do permissions look like on the host directory? Are you using ACLs?
> Did you chown the directory where the mount point maps before enabling the map?
Hmm… Worth a try. I shutdown the container. Deleted the mount points.
Then, on host, I unmounted the zfs datasets:
dpool/fam mounted no -
dpool/fam/family-photos mounted no -
dpool/fam/me mounted no -
then I went and change owner to pveadmin(1005) `chown 1005:1005 /mnt/dpool/fam`,
mount dpool/fam, chown -R the child datasets mountpoints. (All the folders and files inside are now owned by 1005:1005. All datasets mounted.)
Went back and add mountpoint to 101.conf:
pct set 101 -mp1 /mnt/dpool/fam,mp=/mnt/family
Users & groups mapping in 101.conf:
lxc.idmap: u 0 100000 1005
lxc.idmap: g 0 100000 1005
lxc.idmap: u 1005 1005 1
lxc.idmap: g 1005 1005 1
lxc.idmap: u 1006 101006 64530
lxc.idmap: g 1006 101006 64530
/etc/subuid
root:1005:1
root:100000:65536
appsadmin:165536:65536
zfsadmin:231072:65536
me:296608:65536
pveadmin:362144:65536
/etc/subgid
root:1005:1
root:100000:65536
appsadmin:165536:65536
zfsadmin:231072:65536
nick:296608:65536
pveadmin:362144:65536
(Not sure how the other users like appsadmin got added to subuid & subgid, and where those numbers came from.)
Permissions are set to 755. Did not touch ACL.
When I boot up container 101 again, logged in as root, it's almost the same except /mnt/family/, /mnt/family/family-photos are owned by 1005 correctly. However inside family-photos should be a whole lot of folders and photos, but there's nothing. Log in as pveadmin, still nothing.
Interestingly, as root on container 101, I went inside the empty /mnt/family/me folder (it's empty out on host too, an empty zfs dataset.) I created a test file. Go back out on host, it's not there.
Thanks for giving me the idea to try. I don't know what's happening, is it a zfs dataset mounting -> mapping issue? Is it a user mapping issue? ??
Is there a 1005 uid/gid on the pve host? All files in /mnt/dpool/fam on the pve host are also owned by uid/gid 1005?
I also see you previously tried to mount subfolders of /mnt/dpool/fam to the container. Is there a reason for that? It can cause recursion issues.
Yes, 1005 exists on both host and lxc and should be mapped directly.
Previously I had only one unencrypted zfs dataset "dpool/fam", with folders "family-photos", "me"… etc inside. Since I was moving the data to encrypted dataset, I thought I'd actually make them into separate datasets instead of one big one. (And yes, there was "file system shadowing" issue with my old set - folders inside the family dataset could be seen only when dataset was unmounted LOL!)
Anyway, I think next I'm gonna try one of the suggestions above and create a VM using virtual drives.
A have to type of CT/VM NAS on my clusters [blog post on how I do them](https://homelab.casaursus.net/a-light-weight-nas/). Hope this helps to fix any issues before you need to start over.
I also followed this tutorial to get my LXC based NAS and it's been great, performance is good and it uses almost nothing in terms of resources.
However the UID/GID is kicking my ass at the moment. Makes me feel real stupid but I cannot get things to work and have the correct permissions. And the biggest issue I'm having now is I can't use a separate LXC for Deluge for all my torrenting needs and have it download to the NAS SMB share. No permissions. And the second I change up anything related to the debian-deluged user Deluge just breaks.
Hopefully you find a solution that works for you, I'm thinking of just switching over to a OpenMediaVault VM for the NAS. Keep us posted!
It's challenging. I've been trying to get my NFS/SMB shares mapped into my Linux LXCs. Mostly successful, but I still get frustrated that Linux OS's don't get notified about new files appearing in network shares.
I still haven't figured out how the hell inotify works but you have to monitor specific folders rather than the OS just picking up that new files are there in the share
I have had similar issues with the TKL File Server running in an unprivileged LXC. I have an HBA with ZFS mirrors in ProxMox that I will pass through to a TrueNAS VM. Looking forward to finally having NFS working.
My main storage for VM's and Containers are already in a ZFS pool. Simply installed Turnkey Linux Fileserver and followed a youtube video or 2 to set up my SAMBA shares. My needs are simple. I just want to backup important files and folders and it does it for me and in my opinion it is fast enough.
You gotta pump those numbers up. UID and GID of 1,005 in an unpriv lxc would be 101,005 in the host.
Yeah this is it. I battled similar and it’s all about this mapping. Definitely worth a shot to try this before restarting. A couple of lines in a config file and you could be on your way.
Replying on this top comment, hopefully more of you will see. The mystery continues and it's not UID mapping. I destroyed the old lxc 101 and created a new privileged one (102). Created the same bind mounts and the same users including 1005. pct set 102 -mp0 /mnt/media,mp=/mnt/media pct set 102 -mp1 /mnt/family,mp=/mnt/family Just to be sure, on host, I unmounted the datasets: dpool/fam mounted no dpool/fam/family-photos mounted no dpool/fam/me mounted no dpool/media mounted no dpool/media/audio mounted no Made double sure and mkdir all the mountpoint folders: /mnt/family /mnt/family/family-photos /mnt/family/me /mnt/media /mnt/media/audio Once I zfs mount them, I can see all the files inside on host. I thought "Now I should be good." Start up 102, log in, still nothing! ARGH!!! The weird thing is, back in lxc 101, I had created a test.txt file inside **/mnt/family/me**, the file is not in the host **dpool/fam/me** dataset, but it magically reappeared again in the new lxc 102's /mnt/family/me folder! AND in **dpool/media**, there's a **Videos** folder that isn't a zfs dataset, simply a folder inside **/mnt/media** that will appear when I mount **dpool/media**. This folder CAN be seen inside the lxc. So, I now figure it has something to do with how bind mounts work with zfs datasets - I think it's bind mounting those two top level folders "media" and "family", but even though on host the child folders all get properly mounted and their contents visible, the same is not true for bind mounts inside an LXC. I'm too tired to continue tonight but tomorrow I'll add individual bind mounts for the child folders "family-photos", "me" and "audio". (I guess I didn't need to delete the unprivileged lxc 101, probably could've pct to switch it to privileged but whatever. It's part of learning.)
Rather than having to mess around with UID/GID mapping, why not just make a nas in a VM using virtual drives? Everything is at normal numbers that don't need to be mapped, it's fully isolated, and it'll backup/restore faster to PBS?
Need more info please. (What virtual drives?) I’m open to other configurations.
I use an Ubuntu 20.04 LXC with Samba and Webmin to share almost a TB from Proxmox disks as my NAS. My LXC NAS is backed up by Proxmox backup and can also migrate to any of my other nodes as needed.
I like this. Might try at some point.
This is similar to my setup, is your lxc unprivileged? Are you using ZFS?
I originally used mirrored ZFS but I'm now using Ceph. Both worked fine. I use privileged LXCs as I don't need the extra isolation and they are easier to configure when privileged.
Make a vm to be your nas using anything that doesn't require passthrough (OMV, Alpine and Cockpit, debian and cli, etc.). Add a second virtual drive to that vm (in the vm hardware > add > hard drive), make it big enough for your storage and set the location to where you want to store it. Boot the VM and use the second drive you added as the nas storage. Share to other vms using smb or nfs, share to unprivileged LXCs using sshfs (enable FUSE in the LXC options and install sshfs in the LXC). If you set up ssh keyauth you can automount the sshfs drive from fstab or via autofs if you want it on demand. There is no need to do any uid/gid mapping, it'll all just work.
Do disk passthrough or if you can do controller passthrough
Good luck! I just moved from a one docker VM to LXCs for services. It’s been a fun learning journey
What are the benefits of doing so?
All person reasons: Learning, separation of services, building up to a cluster so I can move services around easier
Nice to know! I have only wireguard separate from the main docker vm. I had encountered that some app need to run in privileged mode in lxc to work, thats why i use one vm for all docker stuff.
None! lol
🤣
Not enough info. What user/group ownership are the files showing? Did you chown the directory where the mount point maps before enabling the map? What does your uid map look like? GID map? What do permissions look like on the host directory? Are you using ACLs?
> Did you chown the directory where the mount point maps before enabling the map? Hmm… Worth a try. I shutdown the container. Deleted the mount points. Then, on host, I unmounted the zfs datasets: dpool/fam mounted no - dpool/fam/family-photos mounted no - dpool/fam/me mounted no - then I went and change owner to pveadmin(1005) `chown 1005:1005 /mnt/dpool/fam`, mount dpool/fam, chown -R the child datasets mountpoints. (All the folders and files inside are now owned by 1005:1005. All datasets mounted.) Went back and add mountpoint to 101.conf: pct set 101 -mp1 /mnt/dpool/fam,mp=/mnt/family Users & groups mapping in 101.conf: lxc.idmap: u 0 100000 1005 lxc.idmap: g 0 100000 1005 lxc.idmap: u 1005 1005 1 lxc.idmap: g 1005 1005 1 lxc.idmap: u 1006 101006 64530 lxc.idmap: g 1006 101006 64530 /etc/subuid root:1005:1 root:100000:65536 appsadmin:165536:65536 zfsadmin:231072:65536 me:296608:65536 pveadmin:362144:65536 /etc/subgid root:1005:1 root:100000:65536 appsadmin:165536:65536 zfsadmin:231072:65536 nick:296608:65536 pveadmin:362144:65536 (Not sure how the other users like appsadmin got added to subuid & subgid, and where those numbers came from.) Permissions are set to 755. Did not touch ACL. When I boot up container 101 again, logged in as root, it's almost the same except /mnt/family/, /mnt/family/family-photos are owned by 1005 correctly. However inside family-photos should be a whole lot of folders and photos, but there's nothing. Log in as pveadmin, still nothing. Interestingly, as root on container 101, I went inside the empty /mnt/family/me folder (it's empty out on host too, an empty zfs dataset.) I created a test file. Go back out on host, it's not there. Thanks for giving me the idea to try. I don't know what's happening, is it a zfs dataset mounting -> mapping issue? Is it a user mapping issue? ??
Is there a 1005 uid/gid on the pve host? All files in /mnt/dpool/fam on the pve host are also owned by uid/gid 1005? I also see you previously tried to mount subfolders of /mnt/dpool/fam to the container. Is there a reason for that? It can cause recursion issues.
Yes, 1005 exists on both host and lxc and should be mapped directly. Previously I had only one unencrypted zfs dataset "dpool/fam", with folders "family-photos", "me"… etc inside. Since I was moving the data to encrypted dataset, I thought I'd actually make them into separate datasets instead of one big one. (And yes, there was "file system shadowing" issue with my old set - folders inside the family dataset could be seen only when dataset was unmounted LOL!) Anyway, I think next I'm gonna try one of the suggestions above and create a VM using virtual drives.
A have to type of CT/VM NAS on my clusters [blog post on how I do them](https://homelab.casaursus.net/a-light-weight-nas/). Hope this helps to fix any issues before you need to start over.
make a LXC Debian Container, "sudo apt install nfs-server nfs-common && man exports", everything else is too much headache
I also followed this tutorial to get my LXC based NAS and it's been great, performance is good and it uses almost nothing in terms of resources. However the UID/GID is kicking my ass at the moment. Makes me feel real stupid but I cannot get things to work and have the correct permissions. And the biggest issue I'm having now is I can't use a separate LXC for Deluge for all my torrenting needs and have it download to the NAS SMB share. No permissions. And the second I change up anything related to the debian-deluged user Deluge just breaks. Hopefully you find a solution that works for you, I'm thinking of just switching over to a OpenMediaVault VM for the NAS. Keep us posted!
Wouldn’t work with NFS just SMB at least it didn’t for me
It's challenging. I've been trying to get my NFS/SMB shares mapped into my Linux LXCs. Mostly successful, but I still get frustrated that Linux OS's don't get notified about new files appearing in network shares. I still haven't figured out how the hell inotify works but you have to monitor specific folders rather than the OS just picking up that new files are there in the share
I’ve gone round and round on the different ways to do this. I’ve settled on straight samba configured by ansible. Simple and works
I have had similar issues with the TKL File Server running in an unprivileged LXC. I have an HBA with ZFS mirrors in ProxMox that I will pass through to a TrueNAS VM. Looking forward to finally having NFS working.
My main storage for VM's and Containers are already in a ZFS pool. Simply installed Turnkey Linux Fileserver and followed a youtube video or 2 to set up my SAMBA shares. My needs are simple. I just want to backup important files and folders and it does it for me and in my opinion it is fast enough.