The YGOR data enrichment tool is still under development.
Use at your own risk!


What is Ygor?

Ygor is a small tool for data enrichment of KBart data. Bibliographical data in KBart format (Knowledge Base and Related Tools)
will be enriched automatically with additional data by external web services such as ZDB and EZB. This process needs appropriate
title identifiers such as e-ISSN, p-ISSN or the zdb id of the German Union Catalogue of Serials (ZDB).

Via the Knowledge Base coreferencing API, the enriched data can be imported into a connected Knowledge Base.

To start processing a KBart file, please select "Data enrichment" in the menu.



0.96

- Enhancements:
...Add updates-per-line-check with last_changed
...Enable timestamped URL updates, both initially and real updates
...Give direct upload feedback for GOKb-triggered uploads with CompleteProcessingThread and UploadJobFrame for GOKb updates
...Allow unknown encoding for KBart URLs
...Switch zdb and ezb update check from package to source
...Set tipp.title.name to tipp.name when missing and postprocess accordingly
...Allow missing name of title (due to omitted API enrichment)
...Use publication_title for title.name when missing API enrichment
...Add endpoint checkAvailability
...Set session duration individually per action
...Finalize addOnly functionality for GOKb triggered package updates
...Add error checks for getting package in processGokbPackage
...Tweak timing of GOKb Ygor update processes
...Support redirects for KBart URLs
...Make Curatory Group selection required
...Suppress pretty Json printing for enrichments with more than 1000 records
- Fixes and stuff:
...Exchange CSV parser to make escaped processing working
...Add some informative logging
...Fix message handling
...Fix params handling & result mapping
...Fix record validation (record name)
...Fix empty field value following empty KBart cell

0.95

- Enhancements:
...Tune performance
...Add endpoint /enrichment/processGokbPackage
...Add endpoint /enrichment/getStatus
...Add Download button for package with integrated title data
...Add Send button for package with integrated title data
...Add handling for GOKb response 400
...Enable getting GOKb jobId with token-based authentification
...Send packages with integrated title data asynchronously
...Add parameter isUpdate
...Assign series, listprices and subjectArea to TIPP output
...Guess encoding from file content in case there is no URL information
...i18nize encoding error messages
...Refine KBart URL encoding detection
...Enable connection to KBart URL where server needs a user agent
...Implement updates with variable URLs
...Set ZDB and EZB enrichment to off per default
- Fixes:
...Fixing package upload from new GOKb UI
...Fix locale processing on uploadUrl
...Display statistic tables with length 1
...Prevent out of bounds when filtering best API match
...Catch NPE occuring on missing prices
...Fix send package
...Fix update processing

0.94

- Functional and configurational changes:
...Add token-based requests for new GOKb-UI
...Add upload from URL function
...Add option for daily KBart-URL update checks
...Add automatic updates function
...Add complete processing function without UI interaction
...Add combined Tipp & Title structure for GOKb export in case complete processing
...Add locale information when sending GOKb request
...Remove manual delimiter selection
...Remove error flag for publicationType when field has been fixed
...Configure additional KBart fields for series, listprice and subject area
...Normalize publication_type to lower case
...Allow publicationType "other"
...Stop main progress increase at 99% before showing result
...Round values of GOKb upload bar and don't let it increase to 100
...Amend GOKb response status check to catch multiple cases or changes in GOKb
- Documentation & Cleanup:
...Update README.md
...Remove obsolete sessionFolder parameter from update processing
...Refactor package header: remove pkg and package title placeholder
...Remove old German-only version log entries
...Remove stale config entries
...Replace occurences of "KBART" by "KBart"
- Fixes:
...Replace hiding of empty tables by ajax function to avoid empty result page
...Hotfix: Catch NPE on missing coverage_depth
...Hotfix: prevent process aborting on re-calculation of RecordFlag
...Fix YYYY-MM normalization
...Fix error message on missing enrichment
...Fix handling of records missing a title
...Fix display behaviour for erroneous packages
...Remove stale code
...Hotfix: prevent process aborting on reconstructing RecordFlag

0.93

- Enhancements:
...Set ZDB warning panel colour depending on enrichment conditions and record validity
...Show number of records on result list boxes
...Remove KBart fields from mandatory fields list
...Set async=true and addOnly=true for send package requests only
...Remove very stale nominal platform preset list
...Render DataTables truncated from the beginning to reduce rendering effort
...Wait for 750ms before rendering upload status
Fixes:
...Don't create Records from empty KBart lines
...Fix progress amount for package upload
...Fix progress bar after speeding up rendering
...Catch NumberFormatException in statistics show
...Catch error on file upload
...Prevent exception when uploading

0.92

- Enhancements:
...Amend UI to match the design of GOKb
...Show error result details of GOKb package upload
...Re-activate toggle function for upload results sections
...Disable upload buttons after having triggered upload
...Show upload results also after editing records
...Update apache commons validator to match recent TLDs
...Add empty check for date formatting
...Update Grails Tomcat plugin to 8.0.33 (using Tomcat 9)
Fixes:
...Fix statistics table pagination
...Fully qualify IntegrationStatus in child classes to suppress compilation errors
...Fix persisting duplicates
...Hotfix: don't RegEx match empty values in MultiField
...Catch exceptions on empty data fields
...Catch exception on missing value type in normalization process
...Catch exception in Json generation

0.91

- Enhancements:
...Progress bar for titles and package upload
...Allow displaying feedback of multiple uploads
...Enable page refresh on any status of GOKb uploads
...Add 'Add titles only' option for GOKb package upload

0.90

- Enhancements:
...Process data tables sorting on server-side
...Optimize performance of main processing phase
...Optimize performance of GOKb upload
...Implement asynchronous upload to GOKb and add upload progress bar
...Refactor duplicates marks calculation
...Load Records from disk and save Records to disk
...Loading Enrichment without DataContainer
...Truncate long display titles in record tables
...Display identifiers without type
Fixes & system-internal changes:
...Fix loop instruction on server-side DataTables
...Fix raw data export
...Fix raw data upload
...Fix GOKb export after raw data upload
...Improve error logging
...Remove stale code

0.89

- Refactoring and enhancements:
...Improve field naming and enrichment order
...Make ZDB enrichment optional
...Set identifiers dynamically for (e)ISBN vs. (e)ISSN
...Display GOKb response as accordeon
...Set obligatory input fields as 'required'
...Show package name on results main page
...Show 'missing' for record when lacking title
...Re-arrange buttons on statistics list view
...Implement flag for order of date issue online
...Normalize ZDB date format [YYYY]-
...Catch expired session
...Add null checks to identifier postprocessing
Fixes:
...Optimizations and fixes in date normalization
...Fix persistence of RecordFlag
...Fixes and optimizations in Record, RecordFlag and StatisticController
...Re-implement isil processing
...Fix setting identifiers type for package export
...Fix BOM deletion

0.88

- Refactoring and enhancements:
...Change curatory groups UI element to dropdown and remove second curatory group
...Add internal package identifier identifier
...Add flags to Record to mark identifier mismatches, classify records accordingly and show flags on edit view
...Put production DB into application memory (since it remains empty anyway)
...Add modal for GOKb upload response (refactoring)
...Catch HTTP 401 when exporting to GOKb
...Allow date format YYYY-MM
...Add EZB ID validation
...Automatically exclude monographs from ZDB enrichment
...Get publication type from KBart file and set status colors accordingly
...Add new warning label (missing ZDB alignment)
...Add record from KBart also if all its ID fields are empty
...Show field's displayName instead of ygorFieldKey on edit page
...Add configuration for shortcode (ZDB field 033A:p)
...Add title ramification (ZDB field 021C:l)
Fixes:
...Handle platform and provider to result
...Stop backend integration on 'Stop processing'
...Change GOKb export of curatory group(s) to array (refactoring)

0.87

- Refactoring:
...Mark duplicates as warnings
...Mark records missing title URL as warnings
...Remove URL-prependix in DOI normalization in title_id
...Add gokb_uuid as new KBart field
...Accept date time spans (YYYY-YYYY) in history events
...Remove curatory group placeholders from UI
Fixes:
...Prevent upload failure on ZDB-ID case difference ('x' vs. 'X')
...Remove historyEvents data stub for invalid historyEvents
...namespace titleId selection in case of re-upload
...export / re-import Curatory Groups

0.86

- Refactoring:
...Hide non relevant fields from statistics edit view
...Record validation is 3-state (red, yellow, green)
...Introduce flags per Json field config to indicate effect of (in-) validity
...Don't export any red (invalid) records
...Export and import raw data as a .zip with multiple Json files
...Replace 'pISSN' by 'ISSN' in GOKb export
...Put all books into green statistics box if valid
...Use ISSN and ISBN patterns in all print_identifier and online_identifier validations
...Allow ISBN-13 validation without hyphens
...Avoid false, empty linked records resulting from org GND id (given as second '039E:0')

0.85

- Refactoring:
...Optimize KBart errors reporting
...Show extended publication title ('displayTitle') in statistics main view
...Postprocess publication title (with sub title and variation)
...Get publication title variation (025@:a) as separate entry
...Add history event title from SRU (039E:Y)
...Process 4-digit SRU history event dates (039E:H)
...Fix: delete old history events (in case of successive titles export)
...Fix: get already existing linked records

0.84

- Refactoring:
...Pre-select title ID in case of re-uploading
...Prevent stacktrace on UI
...Query ZDB API for linked history events during main processing phase

0.83

- Refactoring:
...Get additional identifier values from ZDB API
...Allow multiple entries for same field
...Export historyEvents to GOKb if they are valid
...Improve coverage validation: check number of entries
...Catch error occuring on multiple uploads of same file
...Remove previous, stale classes (bridges, connectors, interfaces)
...Add publisher history place to fields configuration
...Remove invalid history events from GOKb (titles) export (#160)
...Skip process page 4 and show statistics page directly.
...Change source order for publisherName
...Cleanup and small-fix Statistics
...Small style fixes
...Save edited data to files and cleanup file handling
...Remove main menu entry 'Configuration'
...Adjust style of statistics detail table
...Add ZDB match alert panel on edit statistics page
...Show ZDB Api link symbols only when link exists
Fixes:
...Fix ZDB ID validation
...Remove italic words from GOKb send buttons

0.82

- Refactoring:
...Sorting and search function for statistic tables
...Fix: allow empty package name on raw json upload
...Adjust style of 'Info' header
...Validate ISSNs with trailing 'X' and normalize trailing 'x' to 'X'

0.81

- version.081.0

0.80

- version.080.0

0.79

- version.079.0

0.78

- version.078.1

0.77

- version.077.1

0.76

- version.076.1

0.75

- version.075.1

0.74

- version.074.1

0.73

- version.073.1

0.72

- version.072.1

0.71

- version.071.1

0.70

- version.070.1

0.60

- version.060.1

0.59

- version.059.1

0.58

- version.058.1

0.57

- version.057.1

0.56

- version.056.1

0.55

- version.055.1

0.54

- version.054.1

0.53

- Feature: Internationalisation and English language support