WIP: Adapter goes NodeJS

[georg-schwarz]

Open points

• Replace knex with our own postgres implementation
• Code styling and clean-up (Linter)

DONT MERGE YET!

[georg-schwarz]

ods/adapter/src/adapter/importer/HttpImporter.ts

Lines 102 to 118 in 4fe866f

	// The old impl retrieved data as byte array and then converted using encoding:
	// Return new String(rawResponse, Charset.forName((String) parameters.get("encoding")));
	// Unfortunately there does not seem to be a universal method .toString(encoding?: string) in javascript
	const result = await axios
	.get(uri, { responseEncoding: encoding })
	.catch((error: AxiosError) => {
	if (error.response) {
	console.log(error.response);
	throw new Error('Could not Fetch from URI:' + uri);
	}
	throw new ImporterParameterError('Could not Fetch from URI:' + uri);
	});
	// Check if data is object/array -> return stringified (because this method returns string)
	if (typeof result.data === 'object' || Array.isArray(result.data)) {
	return JSON.stringify(result.data);
	}
	return result.data as string;

For this one I wanted to ask, whether somebody knows a better way, because I also do not know one.
The old Java impl got the response as a byte[] and then created a new String from it using the passed encoding:

ods/adapter/src/main/java/org/jvalue/ods/adapterservice/adapter/importer/HttpImporter.java

Lines 59 to 61 in 843943f

	URI uri = URI.create(location);
	byte[] rawResponse = restTemplate.getForEntity(uri, byte[].class).getBody();
	return new String(rawResponse, Charset.forName((String) parameters.get("encoding")));

I think you would need to use fetch API instead of Axios... There you can get the byte stream and interpret based on encoding.

[georg-schwarz]

This is the part that was rewritten from the old impl and I have kept it. Not sure if we want to rewrite it again to match old Java impl.

ods/adapter/src/datasource/services/dataImportTriggerService.ts

Lines 84 to 88 in 4fe866f

	// This is 'new' solution (instead of overriding url -> override params here and url in importer)
	datasource.protocol.parameters.defaultParameters = replacementParameters;
	const parameters = {
	...datasource.protocol.parameters,
	};

ods/adapter/src/adapter/importer/HttpImporter.ts

Lines 86 to 100 in 4fe866f

	// Originally this was done in dataImportTrigger during creation of AdapterConfig
	// -> RuntimeParameter are used, because the defaultParams are overriden during AdapterConfig instead
	if (parameters.defaultParameters !== undefined) {
	const defaultParameters = parameters.defaultParameters as Record<
	string,
	unknown
	>;
	const keys = Object.keys(defaultParameters);
	for (const entry of keys) {
	const value = defaultParameters[entry] as string;
	const regex = new RegExp('{' + entry + '}', 'g');
	uri = uri.replace(regex, value);
	}
	}

I think the old solution was better. The importer should not know that there are differences in parameters but just blindly execute what it gets as input.

[georg-schwarz]

@f3l1x98 I refactored everything except the two open points I replied to. Would be nice if you could take care of them.
CI is currently not working somehow for Adapter integration tests.. doesn't get the exit code somehow.

[f3l1x98]

ods/adapter/src/adapter/importer/HttpImporter.ts

Lines 102 to 118 in 4fe866f

	// The old impl retrieved data as byte array and then converted using encoding:
	// Return new String(rawResponse, Charset.forName((String) parameters.get("encoding")));
	// Unfortunately there does not seem to be a universal method .toString(encoding?: string) in javascript
	const result = await axios
	.get(uri, { responseEncoding: encoding })
	.catch((error: AxiosError) => {
	if (error.response) {
	console.log(error.response);
	throw new Error('Could not Fetch from URI:' + uri);
	}
	throw new ImporterParameterError('Could not Fetch from URI:' + uri);
	});
	// Check if data is object/array -> return stringified (because this method returns string)
	if (typeof result.data === 'object' || Array.isArray(result.data)) {
	return JSON.stringify(result.data);
	}
	return result.data as string;

For this one I wanted to ask, whether somebody knows a better way, because I also do not know one.
The old Java impl got the response as a byte[] and then created a new String from it using the passed encoding:

ods/adapter/src/main/java/org/jvalue/ods/adapterservice/adapter/importer/HttpImporter.java

Lines 59 to 61 in 843943f

	URI uri = URI.create(location);
	byte[] rawResponse = restTemplate.getForEntity(uri, byte[].class).getBody();
	return new String(rawResponse, Charset.forName((String) parameters.get("encoding")));

I think you would need to use fetch API instead of Axios... There you can get the byte stream and interpret based on encoding.

This is not quite possible, because the native fetch api is not available in Node.JS before version 18.0.0.
But I have found a way in which axios returns its data as an ArrayBuffer so I have choosen to use this together with the TextDecoder for converting to string:

ods/adapter/src/adapter/importer/HttpImporter.ts

Lines 69 to 82 in d6c6b2e

	const decoder = new TextDecoder(encoding);
	// Fetch as ArrayBuffer
	const result = await axios
	.get(uri, { responseType: 'arraybuffer' })
	.catch((error: AxiosError) => {
	if (error.response) {
	console.log(error.response);
	throw new Error('Could not Fetch from URI:' + uri);
	}
	throw new ImporterParameterError('Could not Fetch from URI:' + uri);
	});
	// Convert ArrayBuffer response to string using encoding
	return decoder.decode(result.data);

[f3l1x98]

@georg-schwarz I have changed it, but the first issue I have solved differently than proposed, so please have a look at the comment above: #408 (comment)
The other issue has been solved:

ods/adapter/src/datasource/DataImportTriggerService.ts

Lines 86 to 87 in d6c6b2e

	// Fill queryParameters in url
	const parameters = this.fillQueryParameters(datasource, runtimeParameters);

[georg-schwarz]

LGTM!

Do you want to tackle the CI issue as well? I can also do it but will get to it end of week soonest.