Return Styles: Pseud0ch, Terminal, Valhalla, NES, Geocities, Blue Moon. Entire thread

Syncing unknown number of files between multiple servers

Name: Anonymous 2018-05-22 19:59

Imagine we have servers A, B, C. Each server has a directory with multiple CSV files where each file may or may not update between syncs by appending rows. Admins of servers A, B, C want to share these files with each other but do not necessarily trust one another. No CSV file will ever exceed 1MB in size and all servers have good internet speeds.

For an added challenge, there may be a variable number of subdirectories containing a variable number of files.

I'm thinking of a number of ways to go about syncing:
(1) vsftpd + curlftpfs: mount the remote server's directory and use simple UNIX tools to check for new and updated files. Anonymous users may only download files.
(2) GIT: the directory with the public files has a repo that is updated automatically between changes, and remote servers can pull the latest updates automatically.
(3) ATOM feeds: each server announces the added records within files or new files to a server-wide changelog; remote servers use existing ATOM libraries to sync changes
(4) RESTful API: a very basic changelog records which files are updated & at what time, and they can be pulled over HTTP. if-modified-since and similar REST patterns are employed

How would /prog/ go about this? It's not for an assignment and not for work -- just for a personal project.

Name: Anonymous 2018-05-31 4:22

SQL database with a FUSE driver that presents tables as CSV files. Translating read / write to select / insert is fun; keeping the data store consistent is not. Fool around with the former and get the latter for free.

Newer Posts
Don't change these.
Name: Email:
Entire Thread Thread List