movemetafs 0.02

movemetafs is a searchable filesystem metadata store for Linux that lets users tag local files (including image, video, audio, and text files) by simply moving the files to a special folder using any file manager.

It also lets users find files by tags using a boolean search query. The original files (and their names) are kept intact. movemetafs doesn't have its own user interface, but it is usable with any file manager.

In the name `movemetafs', `metafs' means filesystem metadata store, and 'move' refers to the most common way tags are added or removed: the user moves the file to be acted on to the `meta/tag/$TAGNAME' or
`meta/untag/$TAGNAME' special folder. When the target folder is such a special folder, the file is not removed from its original location (meta/root/**/*).

movemetafs is similar to LAFS (http://junk.nocrew.org/~stefan/lafs/)
(not tagji 1.1 by Manuel Arriaga). Most important differences:

· movemetafs uses MySQL instead of PostgreSQL (benefits: speedup, easier installation with pts-mysql-local).
· movemetafs doesn't require files to be explicitly added.
· movemetafs cannot list all untagged files quickly.
· movemetafs is written in Perl, so it is quite easy to extend and try out new features.
· movemetafs continues to work when files are renamed.
· movemetafs works with files with more than one hard link.

Here are some key features of "movemetafs":

· Use any file manager to tag files: move the file to the `meta/tag/$TAGNAME' or `meta/tagged/$TAGNAME' folder. The file is not removed from its original folder.
· Use any file manager to untag files: move the file to the `meta/untag/$TAGNAME'. The file is not removed from its original folder.
· Specify search query by changing to the invisible `meta/search/$QUERYSTRING' folder. Results are symbolic links into `meta/root'.
· If you want to search for only a single tag, list the folder `meta/tagged/$TAGNAME'. Results are symbolic links into `meta/root'.
· Alternatively, you untag a file by removing the symlink `meta/tagged/$TAGNAME/$FILENAME'.
· Alternatively, use POSIX extended attributes to retrieve (or even set) the tags associated to a file.
· Attach textual descriptions to files, and read the description once the file is found.
· Use POSIX extended attributes to get and set the textual description of a file.
· Create a recursive POSIX extended attribute dump (of tags and descriptions) from the database very quickly.
· Restore the created POSIX extended attribute dump with `setfattr --restore'.
· Use versatile search query syntax (MySQL fulltext search) with the possiblity of boolean search (i.e. searching for files matching a combination of tags).
· Copy search results to make backups or to create collections.
· Searching is fast, because it uses a fulltext index.
· After installation, movemetafs can be used on an existing filesystem instantly: there is no migration needed to make an existing filesystem usable with movemetafs: data doesn't have to be copied, moved, touched etc.
· Untagged files don't have any negative effect on the speed of movemetafs, even if the filesystem contains millions of files.
· movemetafs nicely survives a system crash: non-cache data is stored in MySQL InnoDB tables (which use journaling), and even if the whole tag database is lost, the original filesystem remains usable without movemetafs.
· Stores tags in twice: once in a fulltext indexed column in a MyISAM table, and once in an InnoDB relational table. Uses the fast MyISAM table for searches, and the InnoDB table for data recovery.
· Treats file names as opaque 8-bit strings, thus it works with any filesystem character set, even when character sets are mixed in the middle of the filename.
· Uses UTF-8 for tag names and descriptions.
· Filenames in search results are automatically made unique when necessary.
· Following the Unix filesystem design, files with multiple hard links share a commond description and a common set of tags. If tags or the description are changed on one name, the changes also apply to other names, too.

Requirements:

· operating system capable of running FUSE (currently Linux and FreeBSD, movemetafs is tested only on Linux >=2.6.18)
· Root privileges are required for some installation steps, but not for usage. movemetafs is just as safe as FUSE itself in a multiuser environment.
· the FUSE libraries >=2.6.1 (/usr/lib/libfuse.so.?) and header files (/usr/include/fuse.h) (e.g. `apt-get install libfuse-dev' on Debian Sarge). Please note that fuse-2.6.0_rc1 is buggy (e.g. rename() always returns 0).
· Perl >=5.8
· the Fuse Perl module (install with `cpan Fuse' as root)
· MySQL server >=4.1 (you don't have to change your existing MySQL server configuration if you use pts-mysql-local). movemetafs is being tested with MySQL server 5.1. Please let me know if it doesn't work with 5.0 or 4.1. Earlier versions of MySQL are not supported by movemetafs.
· a recent MySQL client library (such as /usr/lib/libmysqlclient.so.*) and headers (such as /usr/include/mysql/mysql.h) (e.g. `apt-get install libmysqlclient15-dev' on Debian Sarge)
· the DBD::mysql Perl module
· the FUSE kernel module loaded (e.g. `modprobe fuse' as root on Linux)

Limitations:

· cannot cross filesystem boundaries. This means that tags cannot be added to (or removed from) files not on the carrier filesystem (--root-prefix=). No other restrictions are present when accessing `meta/root'.
· alpha software, ready for local use only
· cannot cross filesystem boundaries
· doesn't survive a mkfs + rsync migration
· tags are lost when the file is copied (use md5sums?)
· no large file support (maximum file size is 2GB limitation of the Fuse Perl module)
· works only with systems with FUSE support (such as Linux)
· stale tags are not removed automatically
· installation and user documentation is incomplete
· not easy to install (i.e. with package)
· no multiuser support yet (i.e. users cannot share their tags)
· only one containing foldr is displayed for search results
· search result symlink might be stale (i.e. it may not point to the correct target file) if the file has been moved outside the control of movemetafs
· FUSE is a little slower than other layers such as Unionfs or Relayfs
· Perl Fuse.pm is a little slower than writing a Fuse module in C
· no logic structuring (such as taxonomy, thesaurus or ontology) and inference
· For files with multiple (hard) links, symlinks in search results point to only one of filenames (usually the oldest) the other filenames are not stored by movemetafs.
· For files with multiple (hard) links, the principal name cannot be easily removed. (But it can be changed.)
· Original POSIX extended attributes and ACLs are not mirrorred.
· Searching is much faster than tagging and untagging.
· Tags cannot be multiple levels deep (i.e. contain `/').
· File descriptions are not searchable.



Product's homepage