How to Sync 7TB or over 1'000'000 Files to Nextcloud Faster!

Need to transfer a large number of files to Nextcloud? In this video, I show a way to sync 7TB (over 1 Million files) as fast as possible to my Nextcloud server without using the slow Nextcloud client!

#Nextcloud #TechGuides

[akhilleusuggo]

*Solution* For those with "permission denied" :
It's normal that you get permission denied, don't change permissions or anything to keep security high.
1- Instead of moving data to the right directory ( with scp or rsync ), instead* copy them to your home dir, or any other dir where you have permissions.
2- Then move ( mv ) that directory with "sudo", or change to root ( sudo -s ) to the nextcloud directory /data
3- Go to the nextcloud root directory ( before the /data ), and change all files permissions to php/apache. sudo chown www-data:www-data -R *

You're done.

[Aa-weuz]

Thank you. This is going to save me a lot of time. I have 3 TBs that I needed to transfer over to my NextCloud server. I tried to set it up with the sync client but that would have taken weeks. Now I can get it done in one night. Also thanks for the tip on using the IP address instead of the URL for the local sync.


Looking forward to more videos on the Nextcloud apps. I've already set up calendar, tasks, and the news app and sync'd them with my phone, but there are so many others to dive into.

[yurt-page]

NC uses it's own implementation of WebDAV protocol written in PHP. So it's slow by design. Instead of SCp which is a file transfer over encrypted SSH you can install a Samba (windows share) and just use it for a local network transfer. But instead you may just configure a WebDAV module for the web server and it will work much faster than NC. Also you'll have an option: use encrypted HTTPS (e.g. TLS/SSL) when uploading from internet and use plain HTTP when uploading from local network.
There is many mobile apps to sync folders over WebDAV.
Personally I just removed NC and configured Lighttpd and mod_dav. This is enough for me.
If you are working with large video files then this would be a good option.

[nspattak]

I did not know there is a command to make next cloud rescan the local filesystem for changes, thank you. For anyone's information, rsync is usually the best tool to use to copy files between machines (even though in this video scp was almost at the limit of the local network)

[CS-ptcj]

I watched in the last days all your ubuntu and nextcloud related videos, you're great!

[Ropt]

Seu video é realmente MUITO IMPORTANTE. Talvez a maioria dos novos usuários que precissam migrar os dados para a núvel não sabem como fazê-lo de forma rápida porque o Nextcloud não permite simplesmente usar um disco externo com uma entrada usb e fazer essa mudança diretamente. Eu instalei o Nextcloud em docker no meu Qnap. No entanto uso Ubuntu e não WIndows e admito que até agora não consegui fazer o que vc fez tão rápido e simples. Acho que um segundo video mais auto-explicativo usando os comandos do terminal Ubuntu apenas agregava mais. Seu video continua a ser o unico que fala sobre o assunto. Eu também pergunto se não existe alternativas mais fáceis e rápidas sem o uso do terminal. Obrigado!

[MichaelWBell]

Hi there, I am currently getting a 504 gateway timeout MOVE error with my Nextcloud installation... Can anyone help me with this?

[uae]

This video saved my life :)'', , , , i saw it year ago, and today We had a bit of an issue in my nextcloud hardware.. so i setup a new nextcloud server and restorted our files as explained, , , , thank you very very very much

[stephencrescenti]

You are an absolute Nextcloud Beast! Keep up the great work and thank you!

[Gzarane]

please how can i transfer my files from a hosting company to nextcloud without losing the host that i'm working with?

[Lmarc]

Hello. I uploaded all my data from my external hard drive to my server using samba. Next, I adjusted all the ownership to www-data:www-data, 755 for folders and 644 for files, and ran the occ command to update everything. The process ended up well and all files and folders were available when accessing Nextcloud from the browser. However, when I ask to my Nextcloud client (running on MacOS) to sync folders with my Nextcloud, an upload of the whole amount of data starts. It seems like the files do not match and the Nexcloud Client try to update them. I am completely lost on this. Any help would be appreciated.

Regards,

Daniel.

[abruschini]

Ive chosen "Sync everything from server" and sync all my files, but now, i want to add a new folder but with this method, but i dont have the "Add Folder Sync Connection" button (just the last files that were synced). Is it possible to add a folder with this method, even thou i selected "Sync Everyhing from server".? Maybe just copying the folder with scp, and then re scanning files with sudo nextcloud.occ files:scan?

[MrAsh-ivtq]

can we make auto synchronization from raspberry to nextcloud? if possible what could be the process!! please explain!!!1

[aeiben]

have a freenas. managed to copy all my files in to the folder i can't get nextcloud.occ function to work.keeps telling me command not found. I have been trying with ssh connection, tried via shell in freenas, tried with iocage console command...always doing the command in nextcloud folder and also tried out. do i need to install something to get nextcloud.occ to work? thanks

[mirkoregina]

Hi, I have a debian-based Q4OS linux distro and I need to connect 4 hard drives with windows and mac backups and I would like to create a server with raid sharing of all hard drives and including a continuous and incremental backup of smartphones too and ipad in the family ... I was thinking about nextcloud, openmediavault or seafile but there would probably be something else missing to perfect everything and for example the search for folders on different file systems and compatibility with android, ipados, mac and windows. also nvidia shield which can act as a nas if it would be enough and without paying plex. Advice? Thank you.

[ELdren.]

scp: Nextcloud/data/admin/files: No such file or directory Why?

[ArnabKhan]

at 4:12 minute my command is :>> sudo -u apache php occ files:scan -all

[progressumintegrated]

I am interested in knowing how to configure the extended storage o be the primary drive on nextcloud running on Digitalocean's droplet. Using Ubuntu 20.

[timohosen]

What about Activities, when you run the command prompt? Is it showing that new files have been added?

[aaronmatthews]

When I try the file scan it gives me the error “sudo: nextcloud.occ: command not found”. I’m running ssh from terminal in Mac. Does that make a difference?