Metadata-Version: 2.1
Name: python-hzb-rdm-fileformats
Version: 0.2
Summary: A file format guessing tool
Home-page: https://codebase.helmholtz.cloud/hzb/research_data_management/python-hzb-rdm-fileformats
Author: Rolf Krahl
Author-email: rolf.krahl@helmholtz-berlin.de
License: Apache-2.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.6
Description-Content-Type: text/x-rst
License-File: LICENSE.txt
Requires-Dist: lxml
Requires-Dist: python-hzb-rdm
Requires-Dist: python-icat
Requires-Dist: setuptools
Provides-Extra: magic
Requires-Dist: python-magic ; extra == 'magic'

A file format guessing tool
===========================

This package provides a tool to inspect a file and guess the file
format according to configurable rules.


System requirements
-------------------

Python:

+ Python 3.6 or newer.

Required library packages:

+ `setuptools`_

+ `lxml`_

+ `python-hzb-rdm`_

+ `python-icat`_

Optional library packages:

+ `python-magic`_

  Needed if you want to use libmagic to identify the file.  Without
  this package, only matching rules based on the file name pattern are
  available.

+ `git-props`_

  This package is used to extract some metadata such as the version
  number out of git, the version control system.  All releases embed
  that metadata in the distribution.  So this package is only needed
  to build out of the plain development source tree as cloned from
  GitHub, but not to build a release distribution.


Configuration file and matching algorithm
-----------------------------------------

:class:`hzb_rdm.fileformats.FormatChecker` provided by this package
needs a file formats configuration file.  A working example is
provided in the source distribution as `etc/fileformats.xml`.  Let's
consider the following simplified example:

.. code-block:: xml

   <?xml version="1.0" encoding="utf-8"?>
   <fileformats>
     <format>
       <pattern>*.nxs</pattern>
       <filetype>Hierarchical Data Format \(version 5\) data</filetype>
       <name>NeXus</name>
     </format>
     <format>
       <filetype>Hierarchical Data Format \(version 5\) data</filetype>
       <name>HDF5</name>
     </format>
     <format>
       <filetype>ASCII text</filetype>
       <name>Text</name>
     </format>
     <format>
       <name>Other</name>
     </format>
   </fileformats>

It is an XML file, essentially a list of `format` elements, each
defining a file format that will be recognized.  For a given file,
these format entries will be tried in order for a match.  The first
match will be used.  Each format element may have the subelements
`pattern`, `filetype`, `name`, and `version`.  The `name` subelement
is required, the other three are optional.

The `pattern` and `filetype` subelements define tests for a match for
a given file.  The value of `pattern`, if present is used to test the
file name using :func:`fnmatch.fnmatch`.  If `filetype` is present,
:func:`magic.from_file` will be called to inspect the content of the
file.  The value of `filetype` is taken as a regular expression to
test the output of :func:`magic.from_file`.  If :mod:`magic` is not
available, the `filetype` subelement will be ignored.  Note that the
tests restrict the matches: if both, `pattern` and `filetype` are
provided for a format, both test must match for the file to match the
format.  If neither is provided, any file matches the format.  Since
the first matching format for a file is used, the most restrictive
rules should be put first, before the less restrictive ones.

For the example above, all files that :mod:`magic` recognizes as HDF5
file and whose file name uses the extension `.nxs` are matched as
`NeXus` files.  All other HDF5 files are considered as generic `HDF5`.
Files that :mod:`magic` recognize as ASCII text files are matched as
`Text`, regardless of the file name.  The last format entry sets a
catch all default: all unknown files are matches as `Other`.

The `name` and `version` subelement set conditions to search for the
corresponding `DatafileFormat` object in ICAT once the format entry
matches a given file.  If `version` is not provided, the
`DatafileFormat` object must be unique by the name alone, if `version`
is provided, it is taken as an additional condition.


Usage
-----

You need to create a :class:`hzb_rdm.fileformats.FormatChecker` object
providing a file formats configuration file.  This object can then be
used to lookup matching `DatafileFormat` objects from ICAT for your
files::

  >>> from pathlib import Path
  >>> from hzb_rdm.fileformats import FormatChecker
  >>> fc = FormatChecker()
  >>> p = Path("e219891.nxs")
  >>> fc.getDatafileFormat(client, p)
  (datafileFormat){
     createId = "root"
     createTime = 2015-06-18 16:22:24+02:00
     id = 1
     modId = "simple/root"
     modTime = 2019-12-02 16:07:53+01:00
     description = "A common data format for neutron, x-ray and muon science"
     name = "NeXus"
     type = "application/x-hdf5"
     version = "N/A"
   }
  >>> p = Path("e219891.hdf5")
  >>> fc.getDatafileFormat(client, p)
  (datafileFormat){
     createId = "simple/root"
     createTime = 2018-07-19 11:01:12+02:00
     id = 9
     modId = "simple/root"
     modTime = 2019-12-02 16:07:53+01:00
     description = "Hierarchical Data Format version 5"
     name = "HDF5"
     type = "application/x-hdf5"
     version = "N/A"
   }
  >>> p = Path("README")
  >>> fc.getDatafileFormat(client, p)
  (datafileFormat){
     createId = "root"
     createTime = 2015-06-18 16:22:24+02:00
     id = 6
     modId = "root"
     modTime = 2015-11-10 08:54:16+01:00
     description = "Plain text file"
     name = "Text"
     type = "text/plain"
     version = "N/A"
   }
  >>> p = Path("e219891-random.dat")
  >>> fc.getDatafileFormat(client, p)
  (datafileFormat){
     createId = "root"
     createTime = 2015-06-18 16:22:24+02:00
     id = 5
     modId = "root"
     modTime = 2015-06-18 16:22:24+02:00
     description = "Unknown file format"
     name = "Other"
     version = "N/A"
   }


Command line interface
----------------------

The package provides a command line interface.  This is mostly useful
for testing, for instance to verify the effect of changes in the file
formats configuration file::

  $ python3 -m hzb_rdm.fileformats -s anon msg.txt
  msg.txt: Text (version N/A)
  $ python3 -m hzb_rdm.fileformats -s anon --fileformats fileformats.xml --validate msg.txt
  fileformats.xml is valid
  msg.txt: Text (version N/A)


Copyright and License
---------------------

Copyright 2013–2026
Helmholtz-Zentrum Berlin für Materialien und Energie GmbH

Licensed under the `Apache License`_, Version 2.0 (the "License"); you
may not use this package except in compliance with the License.

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.  See the License for the specific language governing
permissions and limitations under the License.


.. _setuptools: https://github.com/pypa/setuptools/
.. _lxml: https://lxml.de/
.. _python-hzb-rdm: https://codebase.helmholtz.cloud/hzb/research_data_management/python-hzb-rdm
.. _python-icat: https://github.com/icatproject/python-icat
.. _python-magic: http://github.com/ahupp/python-magic
.. _git-props: https://github.com/RKrahl/git-props
.. _Apache License: https://www.apache.org/licenses/LICENSE-2.0
