DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic is not resolved, but there is a WORKAROUND.
Author Message
jneasy
Participant



Joined: 29 Jan 2012
Posts: 26
Location: Australia
Points: 268

Post Posted: Mon Nov 05, 2018 2:03 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Windows
Hi,

I have been attempting to parse large JSON files (+200MB) that are provided on a daily basis using the schema provided.

I do know that in the Hierarchical stage the assembly editor can infer the schema by just importing the JSON data file,I have been able to parse a file based on this approach. The problem with this is that I have been given no guarantees that the +200MB file fulfills all fields defined in the schema.

My question is has anyone been able to import a JSON schema and use that to parse JSON data?

I have even tried using a simple Person example found at https://json-schema.org/learn/miscellaneous-examples.html

Using the sample data the first parser step produces the following in the Downstream Output Test Data which you can see the firstName, lastName and age items are not being populated with the Name and Age values;
Code:

{
  "$id": "https://example.com/person.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The person's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    }
  }
}<?xml version="1.0" encoding="UTF-8"?><top>
  <InputLinks/>
  <result>
    <root>
      <__24_id>
        <@originalName>$id</@originalName>
      </__24_id>
      <__24_schema>
        <@originalName>$schema</@originalName>
      </__24_schema>
      <@type>object</@type>
      <properties>
        <firstName>
          <@type>object</@type>
          <@@isPresent>false</@@isPresent>
        </firstName>
        <lastName>
          <@type>object</@type>
          <@@isPresent>false</@@isPresent>
        </lastName>
        <age>
          <@type>object</@type>
          <@@isPresent>false</@@isPresent>
        </age>
        <@type>object</@type>
        <@@isPresent>false</@@isPresent>
      </properties>
    </root>
  </result>
</top>


Cheers,
jneasy.
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42834
Location: Denver, CO
Points: 220807

Post Posted: Tue Nov 13, 2018 10:46 pm Reply with quote    Back to top    

Don't like seeing posts without a single reply so here I am... wondering if you made any progress with this.

_________________
-craig

Your brain is full of spiders, you've got garlic in your soul, Mr. Grinch!
I wouldn't touch you With a thirty-nine-and-a-half foot pole!
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3794

Points: 30536

Post Posted: Thu Nov 15, 2018 1:17 pm Reply with quote    Back to top    

JSON Schema has seen some success, but is not widespread --- not like xml schema and its formality. The Hierarchical Stage uses a formal JSON document --- best suggestion is to find a "complete" on ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
jneasy
Participant



Joined: 29 Jan 2012
Posts: 26
Location: Australia
Points: 268

Post Posted: Thu Nov 15, 2018 8:08 pm Reply with quote    Back to top    

@ chulett : No progress so far. My next thought is to generate some test data based on the JSON schema. This is where I run into my next problem, the schema is full of cascading references.

@ eostic : I thought someone would comeback with the trying to find a "complete" JSON file. Ive been working off this premise so far and is mostly working but I think I will need to dummy up a "complete" file to complete all mappings.

Appreciate the help guys!

Im going to mark this topic as work around. Work around being generating a "complete" JSON file.
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3794

Points: 30536

Post Posted: Fri Nov 16, 2018 11:32 am Reply with quote    Back to top    

That's the best approach due to the fact that there is no formal standard for JSON schemas. Note --- Be careful when completing your document to fully represent your arrays...meaning...if you h ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours