Sage-Bionetworks/schematic: v23.9.2

What's Changed implement GE list rule by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/655 Develop initial rule pairs by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/802 Cross Manifest Validation: Value Level Comparisons and Reworked Warning Messages by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/751 Develop schema viz tool tests by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/659 make sorting case insensitive by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/830 Develop fix submit keyerror by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/831 updating the release documentation by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/817 Develop an endpoint to return data types of columns in a manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/815 Develop rule combination by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/766 BugFix: Import dataType from CLI as list not string by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/836 BugFix: Manifest Import - Mask null cells as empty strings instead of NaNs by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/837 BugFix: list like behavior by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/840 Check type of 'url' entry when raising errors by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/844 Fix example data model list validation rule and update "list" in ManifestGenerator to handle both list strict and list like by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/847 add warning to validate_manifest endpoint by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/859 Develop NA string behavior by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/842 Develop blank annotation behavior by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/862 Develop Missing Entity Handling by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/871 rename manifest based on manifest_basename when submit to synapse by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/869 Output errors generated from jsonSchema (when submitting manifest) to the console by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/873 Develop manifest submission error handle by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/879 do not create dir if located in cwd by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/881 Update annotations unit test by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/880 Develop optional rule args by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/843 Containerizing schematic (and update dependencies as needed) by @BrunoGrandePhD in https://github.com/Sage-Bionetworks/schematic/pull/882 add endpoint find_class_specific_properties by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/890 add route get_subgraph_by_edge_type by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/891 create endpoint to return schema as a pickle file by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/892 poetry documentation by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/895 Add support for Schematic API to Dockerfile by @BrunoGrandePhD in https://github.com/Sage-Bionetworks/schematic/pull/887 Create project table manifests, move entities to new project by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/829 add concurrency to github action by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/896 HOTFIX: Mixed type column manifest table upload by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/902 update asset view table endpoint to allow returning a json by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/912 added get_node_range endpoint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/904 added get_property_label endpoint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/905 added node_dependencies endpoint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/906 Allow schema.org schema to be loaded and used by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/903 add example workbook to demo cli functions by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/916 change manifest name for censored manifest when upload by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/931 Add Restriction Flag to Make Synapse Table Method by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/935 Develop Table Uploads: Replace by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/917 allow users to upload csv or json file when submitting manifest using API endpoints by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/908 allow users to upload csv or json file when validating manifest using API endpoints by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/910 Develop table schema fix replacements by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/947 Fixing parsing of table schema parameters on table replace by @milen-sage in https://github.com/Sage-Bionetworks/schematic/pull/945 Develop schema viz tool cors by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/942 Move Schema Visualization to Schematic by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/645 add default max size for columns missing this key by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/952 populate manifest as an excel spreadsheet without sending data to Google api by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/940 Skip Trashed Entities and Entities that "do not exist" by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/985 Reword conditional requirements for the attribute table by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/987 add tests related to manifest get by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/979 fix api endpoint for docker container by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/974 Remove unused libraries by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/991 Remove Manifest Table Submission Dependency on Project Scope when not Validating by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/989 HOTFIX: Extend table replacement sleep duration to ensure synapse deletion operation completes by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/999 Pass config.yml as an environment variable to docker container by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/975 update error messages in synapseStore when manifest_basename key is missing in config.yml by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/997 Added instructions for installing and using schematic and the schematic API docker containers to README by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/953 Develop pdoc by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/957 correct typo: master_basename to manifest_basename in error message by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1005 Fixed typo in conditional requirement for schema viz attribute table by adding quotes by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1003 Added tests for all API endpoints by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/995 Fixed an issue where the DCA would disconnect on manifest submission by casting UUIDs to type string by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1006 Reminded users to run synapse login --rememberMe after schematic init step in ReadMe by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1004 Added the option to populate manifest as an excel spreadsheet to avoid sending metadata to Google APIs by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/994 For schematic APIs, return excel file instead of excel file path by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/988 Emphasized that users should download the config.yml from develop branch in ReadMe by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1007 Removed validity requirement for unrequired attribute entries, allow users to specify validation rule conformity required for entries, fix DCA disconnect caused by JSON Schema validation errors by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1000 Strip hyphens from Synapse annotation keys by @milen-sage in https://github.com/Sage-Bionetworks/schematic/pull/1020 Fixed a bug where in some instances manifests would be uploaded with different display and downloadAs names by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1017 Added an API endpoint to check if a given node is required or not by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1024 added more instructions of running CLI in jupyter notebook by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1031 Added a slack bot to notify successful new releases by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1025 Turned service_account_creds.json as an environment variable and default way of authentication by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1015 Changed base docker image and removed package for security reasons by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1037 Add support for treating date entries as type datetime by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1041 Automatically updated CLI documentation on Readthedocs by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1047 Deprecated token pickle and credentials.json by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1040 Introduce tests to cover current table operations: creation and replacement by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1046 Skip api tests when rule combination tests are run by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1068 Added workflow to deploy schematic docker container in Github container registry by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1062 Remove schematic support for Python v3.7 and v3.8 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1090 Refactor table operations structure in asset store by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1069 Added input_token as a parameter for /manifest/get endpoint to fix credential issues when getting an existing manifest on AWS by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1080 Fixed getProjectManifests function in synapse storage by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1084 Develop api node display names by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1094 Create API endpoint for get_node_validation_rules by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1095 Update schematic dependencies by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1092 Raise errors for wrong schema errors by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1073 Set default of "table_manipulation" as "replace" in API endpoint when users enter None and updated tests by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1115 Update synapseClient dependency and api for manifest table uploads by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1101 set pyopenssl = "^23.0.0" by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1125 added date GE rule by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1103 Implement table upsert feature by using schematic-db by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1081 Add use_schema_label parameter to manifest submission endpoint, separate manifest submission and table upsert tests by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1129 Delete GE checkpoint after completion of GE validation by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1136 Remove try: catch: block from manifest submission command function by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1130 Save all properties that are Included in the domain of a Class by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1134 Display exceptions raised during validation with Great expectations, allow exclusion of upper bound OR lower bound for inRange rule by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1131 Update Documentation - python/package versions and POCs by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1139 Increase buffer size to a higher limit to deal with long token by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1144 lock schematic-db to version 0.0.6 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1145 use try: finally: to delete checkpoint even if running the checkpoint fails or errors out by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1155 Allowed CORS on given routes instead of all routes by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1168 Added restrict rules param to manifest/validate by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1178 Bug Fix: remedy negation of table manipulation specification by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1186 Added an endpoint to check entity type on Synapse and an endpoint to check if an entity is in the asset view by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1078 add restrict rules control to manifest validate by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1189 Added a parameter to control if GE gets used when using manifest/validate endpoint by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1177 Propagate logger level entered in from command line to other schematic submodules by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1180 Add timing of validation operations to DEBUG log by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1181 Standardize validation error format and type by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1183 Added function to calculate and clear cache by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1190 Develop add file only manifest submission option by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1175 Create example data model for single rule benchmarking by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1193 Develop api tests for benchmarking single rule validation performance by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1184 Fixed column headers problem when generating an Excel spreadsheet for getting an existing manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1164 Do not index attribute visualization dataframe when converting to csv for component viz endpoint by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1196 Remove restrictions on rule number and allowed combinations by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1203 Start checking mypy and black by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1204 Add IsNA rule to validation suite | FDS-81 FDS-232 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1200 Add API endpoint to visualize attributes for a specific component by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1195 Added functionality to download a manifest by using the manifest id by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1192 Introduce workflow for API tests by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1208 Remove requirement for .synapseConfig file to use upsert feature by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1207 Parallelize operations in load_df by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1100 Id Column - fix bug preventing table updates by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1214 Id Column - fix bug where there would be two columns with uuid values by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1215 Merge AWS deployment branch to develop by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1142 Id Column - fix bug preventing upserts by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1216 Downloaded manifests to temporary folders on AWS by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1210 FDS-293 Fix column mismatch for Excel File Based Manifest Generation by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1213 Feature fds 361 pylint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1212 Changed workflow to run poetry version 1.2.0 by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1222 remove breakpoint from command by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1224 Restrict typing-extensions package to versions before v4.6.0 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1228 Make aws group dependencies optional and restrict typing_extensions package version by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1227 Change Uuid column name to Id by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1211 Avoided publishing minor releases to Docker Hub by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1231 Remove lock on schematic_db version - FDS-287 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1232 feat: added nginx to encrypt the communication between ALB and API by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1202 Installed schematic in docker file by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1230 fix: Fixed unit test in test_manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1233 Fix formatting issues when pulling data from Synapse to Excel Manifest by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1234 fix: fixed tests in test_api.py by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1225 Added test manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1238 [Bug fix]: Tried install pdoc again in github action workflow pdoc.yml by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1239 Replaced synapse API calls with synapse Python client call by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1235 Feat: allowed production tag, staging tag, and manual run on existing tags to trigger building docker images for AWS deployment by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1241 Added hide-blanks parameter when submitting manifest through API by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1242 Revert "Feat: allowed production tag, staging tag, and manual run on existing tags to trigger building docker images for AWS deployment" by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1243 feat: allowed production tag, staging tag, and manual run on existing tags to trigger building docker images for AWS deployment by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1244 [Bug fix]: Fix regex in docker build workflow by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1246 Revert "[Bug fix]: Fix regex in docker build workflow" by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1248 [Bug fix]: Fix regex in docker build workflow by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1249 Feature fds 273 coniguration by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1219 Expose strict_validation option for manifest/generate endpoint by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1253 Optimize performance in table upsert backwards compatibility scenarios by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1229 Fix df_utils/load_df so it more accurately captures integers to prevent Regex Errors. by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1240 Use FAIR Data service desk for issues by @afwillia in https://github.com/Sage-Bionetworks/schematic/pull/1257 Update Black to 2023 version: 23.7.0 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1262 Update mypy to latest version by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1263 Updated cli help text for parameter json_schema by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1264 Modified Nginx parameters to fix submission issue by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1255 modify tag by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1251 BugFix: Fixed an issue where files on synapse were not being annotated correctly by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1254 [bug fix] Fixed errors when calling get_empty_manifest by @linglp in https://github.com/Sage-Bionetwork