performBackgroundTask not saving when app in background Swift

I'm using Xcode 9 and Swift 4. I start a download of multiple JSON files when the application is in the foreground. The application then parses these files and saves them to CoreData. This works well when the application is in the foreground. However, if the application is in the background, the files still download correctly, but the data is not parsed and saved to CoreData. It's only when the user returns to the foreground that the parsing and saving of data continues.

I have Background Modes turned on - Background Fetch and Remote notifications.

I have around 10 functions that are similar to the one below in which it processes the JSON files concurrently:

func populateStuff(json: JSONDictionary) -> Void {     let results = json["result"] as! JSONDictionary     let stuffData = results["Stuff"] as! [JSONDictionary]      let persistentContainer = getPersistentContainer()      persistentContainer.performBackgroundTask { (context) in         for stuff in stuffData {             let newStuff = Stuff(context: context)             newStuff.stuffCode = stuff["Code"] as? String             newStuff.stuffDescription = stuff["Description"] as? String              do {                 try context.save()             } catch {                 fatalError("Failure to save context: \(error)")             }         }     } }  func getPersistentContainer() -> NSPersistentContainer {     let persistentContainer = NSPersistentContainer(name: "MyProjectName")     persistentContainer.loadPersistentStores { (_, error) in         if let error = error {             fatalError("Failed to load core data stack: \(error.localizedDescription)")         }     }     persistentContainer.viewContext.automaticallyMergesChangesFromParent = true     persistentContainer.viewContext.mergePolicy = NSMergePolicy.mergeByPropertyObjectTrump     return persistentContainer } 

Can anyone advise me on why this might happen and how to over come this?

TIA

1 Answers

Answers 1

Use the beginBackgroundTaskWithName:expirationHandler: method:

func populateStuff(json: JSONDictionary) -> Void {      // Perform the task on a background queue.     DispatchQueue.global().async {          // Request the task assertion and save the ID.         self.backgroundTaskID = UIApplication.shared.beginBackgroundTask (withName: "Finish Network Tasks") {          // End the task if time expires.         UIApplication.shared.endBackgroundTask(self.backgroundTaskID!)         self.backgroundTaskID = UIBackgroundTaskInvalid     }      // Parse the json files      let results = json["result"] as! JSONDictionary     let stuffData = results["Stuff"] as! [JSONDictionary]      let persistentContainer = getPersistentContainer()      persistentContainer.performBackgroundTask { (context) in         for stuff in stuffData {             let newStuff = Stuff(context: context)             newStuff.stuffCode = stuff["Code"] as? String             newStuff.stuffDescription = stuff["Description"] as? String              do {                 try context.save()             } catch {                 fatalError("Failure to save context: \(error)")             }         }     }      // End the task assertion.     UIApplication.shared.endBackgroundTask(self.backgroundTaskID!)     self.backgroundTaskID = UIBackgroundTaskInvalid } 

Calling this method gives you extra time to perform important tasks. Notice the use of endBackgroundTask: method right after the task is done. It lets the system know that you are done. If you do not end your tasks in a timely manner, the system terminates your app.

Read More

A circular reference was detected while serializing an object of type when performing ajax call

On my view, I am using a Viewmodel, and I have a form that has only one textbox that accepts dates (not a part of the viewmodel) and 3 tables. By default on page load.. the tables are populated with data based on today's date (you can see that in the controller code below), but if a user selects a date and clicks the search button then I want the tables data to be changed without a page refresh based on the date they selected.

@using (Html.BeginForm()) {     <div class="form-group mb-3 mt-3" style="margin-right: -1.3%;">         <div class="input-group col-md-3 offset-md-9">             @Html.TextBox("detailsDate", null, new { id = "Details-Date", @class = "form-control datetimepicker" })             <div class="input-group-append">                 <button id="Details-Date-Btn" type="submit" class="btn btn-outline-primary"><span class="fa fa-search"></span></button>             </div>         </div>     </div> } 

What I am trying to do is if a user selects and date and hits the search button.. I would like the page to not refresh and the tables data have been changed based on the date. As of right now I am getting:

A circular reference was detected while serializing an object of type 'System.Data.Entity.DynamicProxies.tbl_WeighAssc_8AA7AB5F9DAB261D5142F1D5F5BA6705A588A5AAD2D369FBD4B4BC1BBE0487D4'.

Viewmodel

public class PersonnelDetailsVm {     private static ConnectionString db = new ConnectionString();     public PersonnelDetailsVm()     {         CurrentWeekDates = new List<DateTime>();         WeighAssociations = new List<tbl_WeighAssc>();         ArrestAssociations = new List<tbl_TEUArrestAssc>();         InspectionAssociations = new List<tblTEUInspectionAssc>();     }     public string IBM { get; set; }      [Display(Name = "Name")]     public string UserName { get; set; }      public bool Active { get; set; }      public List<DateTime> CurrentWeekDates { get; set; }     public List<tbl_WeighAssc> WeighAssociations { get; set; }     public List<tbl_TEUArrestAssc> ArrestAssociations { get; set; }     public List<tblTEUInspectionAssc> InspectionAssociations { get; set; }     public List<code_WeighLocation> WeighLocations => db.code_WeighLocation.ToList();     public List<code_ArrestType> ArrestTypes => db.code_ArrestType.ToList();     public List<code_InspectionLevel> InspectionLevels => db.code_InspectionLevel.ToList(); } 

---

Ajax:

// Submission //var redirectUrl = '@Url.Action("Index", "Personnels")'; var settings = {}; settings.baseUri = '@Request.ApplicationPath'; var infoGetUrl = ""; if (settings.baseUri === "/AppName") {     infoGetUrl = settings.baseUri + "/Personnels/Details/"; } else {     infoGetUrl = settings.baseUri + "Personnels/Details/"; }  $("#Details-Date-Btn").click(function() {     $.ajax({         url: infoGetUrl,         method: "POST",         data: $("form").serialize(),         success: function(response) {             console.log("success");             $("body").html(response);         },         error: function(jqXHR, textStatus, errorThrown) {             console.log(jqXHR);         }     }); }); 

---

Controller:

public ActionResult Details(string id, string detailsDate) {     if (id == null)     {         return new HttpStatusCodeResult(HttpStatusCode.BadRequest);     }      tblPersonnel tblPersonnel = db.tblPersonnels.Find(id);      if (tblPersonnel == null)     {         return HttpNotFound();     }      Mapper.Initialize(config => config.CreateMap<tblPersonnel, PersonnelDetailsVm>());     PersonnelDetailsVm person = Mapper.Map<tblPersonnel, PersonnelDetailsVm>(tblPersonnel);      var employeeData = EmployeeData.GetEmployee(person.IBM);      person.UserName =         $"{ConvertRankAbbr.Conversion(employeeData.Rank_Position)} {employeeData.FirstName} {employeeData.LastName}";      if (string.IsNullOrWhiteSpace(detailsDate))     {         var startOfWeek = DateTime.Today.AddDays((int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek -                                                  (int)DateTime.Today.DayOfWeek);         person.CurrentWeekDates = Enumerable.Range(0, 7).Select(i => startOfWeek.AddDays(i)).ToList();         var teuFormIds = db.tbl_TEUForm             .Where(x => person.CurrentWeekDates.Contains(x.EventDate) && x.PersonnelIBM == person.IBM).Select(t => t.Id).ToList();          person.WeighAssociations = db.tbl_WeighAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.ArrestAssociations = db.tbl_TEUArrestAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.InspectionAssociations =             db.tblTEUInspectionAsscs.Where(x => teuFormIds.Contains(x.TEUId)).ToList();           return View(person);      }     else     {         var paramDate = DateTime.ParseExact(detailsDate, "MM/dd/yyyy", CultureInfo.CurrentCulture);          var startOfWeek = paramDate.AddDays((int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek -                                                  (int)paramDate.DayOfWeek);         person.CurrentWeekDates = Enumerable.Range(0, 7).Select(i => startOfWeek.AddDays(i)).ToList();         var teuFormIds = db.tbl_TEUForm             .Where(x => person.CurrentWeekDates.Contains(x.EventDate) && x.PersonnelIBM == person.IBM).Select(t => t.Id).ToList();          person.WeighAssociations = db.tbl_WeighAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.ArrestAssociations = db.tbl_TEUArrestAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.InspectionAssociations =             db.tblTEUInspectionAsscs.Where(x => teuFormIds.Contains(x.TEUId)).ToList();          return Json(person, JsonRequestBehavior.AllowGet);     }  } 

---

So, if the actionresult's paramets detailsDate is not null, then it goes into the else statement which returns a JSON object. When debugging this goes through and when the view is returned I am receiving the error I posted above.

Is there a way to replace the model in the view with what I'm returning from the ajax call so the tables can be based on the right date without a page refresh?

Any help is greatly appreciated.

UPDATE

Based on answer's below I have edited the else statement in my controller method to:

Controller

else {     var paramDate = DateTime.ParseExact(detailsDate, "MM/dd/yyyy", CultureInfo.CurrentCulture);      var startOfWeek = paramDate.AddDays((int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek -                                              (int)paramDate.DayOfWeek);     person.CurrentWeekDates = Enumerable.Range(0, 7).Select(i => startOfWeek.AddDays(i)).ToList();     var teuFormIds = db.tbl_TEUForm         .Where(x => person.CurrentWeekDates.Contains(x.EventDate) && x.PersonnelIBM == person.IBM).Select(t => t.Id).ToList();      person.WeighAssociations = db.tbl_WeighAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();     person.ArrestAssociations = db.tbl_TEUArrestAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();     person.InspectionAssociations =         db.tblTEUInspectionAsscs.Where(x => teuFormIds.Contains(x.TEUId)).ToList();      JsonConvert.DefaultSettings = () => new JsonSerializerSettings()     {         PreserveReferencesHandling = PreserveReferencesHandling.All,         ReferenceLoopHandling = ReferenceLoopHandling.Ignore     };      var jsonStr = JsonConvert.SerializeObject(person);      return Json(jsonStr, "text/plain"); } 

My jQuery/Ajax is still the same:

$("#Details-Date-Btn").click(function() {     $.ajax({         url: infoGetUrl,         data: $("form").serialize(),         success: function(response) {             console.log("success");             console.log(response);             $("body").html(response);         },         error: function(jqXHR, textStatus, errorThrown) {             console.log(jqXHR);         }     }); }); 

But now, when the date is selected I am being returned to a page that shows the Json like a plain text file and losing the HTML and CSS like a normal view.

Here is what I am being returned when a date is selected and the button is clicked.

Also, when I check the console when I select a date and click the button for that date to be sent to the controller I am seeing this:

UPDATE 2

Here is one of my tables.. the other ones are the same setup:

<table class="table table-bordered">     <thead>         <tr>             <th></th>             @foreach (var date in Model.CurrentWeekDates)             {                 <th class="text-center">@date.ToString("ddd") <br /> @date.ToShortDateString()</th>             }                 <th class="text-center table-success">Total For Week</th>         </tr>     </thead>     <tbody>         @foreach (var weighLocation in Model.WeighLocations)         {             <tr class="text-center">                 <td class="table-dark">@weighLocation.Weigh_Location</td>                 @foreach (var date in Model.CurrentWeekDates)                 {                     if (Model.WeighAssociations.Any(x => x.tbl_TEUForm.EventDate == date && x.WeighLocationId == weighLocation.ID))                     {                         <td>@Model.WeighAssociations.Single(x => x.tbl_TEUForm.EventDate == date && x.WeighLocationId == weighLocation.ID).OccurenceCount</td>                     }                     else                     {                         <td>0</td>                     }                  }                 <td class="table-success font-weight-bold">@Model.WeighAssociations.Where(x => x.WeighLocationId == weighLocation.ID).Sum(x => x.OccurenceCount)</td>             </tr>         }     </tbody> </table> 

6 Answers

Answers 1

As far as I can see from your problem, to fix it you can do following steps:

1- View

Add a partial view (_Detail.cshtml)

You need a partial view like _Detail that includes your table like this:

@model PersonnelDetailsVm    <table class="table table-bordered">     <thead>         <tr>             <th></th>             @foreach (var date in Model.CurrentWeekDates)             {                 <th class="text-center">@date.ToString("ddd") <br /> @date.ToShortDateString()</th>             }                 <th class="text-center table-success">Total For Week</th>         </tr>     </thead>     <tbody>         @foreach (var weighLocation in Model.WeighLocations)         {             <tr class="text-center">                 <td class="table-dark">@weighLocation.Weigh_Location</td>                 @foreach (var date in Model.CurrentWeekDates)                 {                     if (Model.WeighAssociations.Any(x => x.tbl_TEUForm.EventDate == date && x.WeighLocationId == weighLocation.ID))                     {                         <td>@Model.WeighAssociations.Single(x => x.tbl_TEUForm.EventDate == date && x.WeighLocationId == weighLocation.ID).OccurenceCount</td>                     }                     else                     {                         <td>0</td>                     }                  }                 <td class="table-success font-weight-bold">@Model.WeighAssociations.Where(x => x.WeighLocationId == weighLocation.ID).Sum(x => x.OccurenceCount)</td>             </tr>         }     </tbody> </table> 

2- Controller

Return the partial view

You should fill the model in your controller and pass it to the partial view.

public ActionResult Details(string id, string detailsDate) {     if (id == null)     {         return new HttpStatusCodeResult(HttpStatusCode.BadRequest);     }      tblPersonnel tblPersonnel = db.tblPersonnels.Find(id);      if (tblPersonnel == null)     {         return HttpNotFound();     }      Mapper.Initialize(config => config.CreateMap<tblPersonnel, PersonnelDetailsVm>());     PersonnelDetailsVm person = Mapper.Map<tblPersonnel, PersonnelDetailsVm>(tblPersonnel);      var employeeData = EmployeeData.GetEmployee(person.IBM);      person.UserName =         $"{ConvertRankAbbr.Conversion(employeeData.Rank_Position)} {employeeData.FirstName} {employeeData.LastName}";      if (string.IsNullOrWhiteSpace(detailsDate))     {         var startOfWeek = DateTime.Today.AddDays((int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek -                                                  (int)DateTime.Today.DayOfWeek);         person.CurrentWeekDates = Enumerable.Range(0, 7).Select(i => startOfWeek.AddDays(i)).ToList();         var teuFormIds = db.tbl_TEUForm             .Where(x => person.CurrentWeekDates.Contains(x.EventDate) && x.PersonnelIBM == person.IBM).Select(t => t.Id).ToList();          person.WeighAssociations = db.tbl_WeighAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.ArrestAssociations = db.tbl_TEUArrestAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.InspectionAssociations =             db.tblTEUInspectionAsscs.Where(x => teuFormIds.Contains(x.TEUId)).ToList();           // return View(person);         }     else     {         var paramDate = DateTime.ParseExact(detailsDate, "MM/dd/yyyy", CultureInfo.CurrentCulture);          var startOfWeek = paramDate.AddDays((int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek -                                                  (int)paramDate.DayOfWeek);         person.CurrentWeekDates = Enumerable.Range(0, 7).Select(i => startOfWeek.AddDays(i)).ToList();         var teuFormIds = db.tbl_TEUForm             .Where(x => person.CurrentWeekDates.Contains(x.EventDate) && x.PersonnelIBM == person.IBM).Select(t => t.Id).ToList();          person.WeighAssociations = db.tbl_WeighAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.ArrestAssociations = db.tbl_TEUArrestAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList();         person.InspectionAssociations =             db.tblTEUInspectionAsscs.Where(x => teuFormIds.Contains(x.TEUId)).ToList();          // return Json(person, JsonRequestBehavior.AllowGet);     }      // return PartialView with the person model     return PartialView("_Detail", person);  } 

As you can see in the above code, you should comment the two lines in your code:

// return View(person);  // return Json(person, JsonRequestBehavior.AllowGet); 

3- Ajax call

Get the partial view and fill form by it

You don't any changes in ajax call and you can do it like this:

$("#Details-Date-Btn").click(function() {     $.ajax({         url: infoGetUrl,         method: "POST",         data: $("form").serialize(),         success: function(response) {             console.log("success");             $("body").html(response);         },         error: function(jqXHR, textStatus, errorThrown) {             console.log(jqXHR);         }     }); }); 

response in this way is a html that comes from partial view and it has all classes as you designed it.

Answers 2

That error message means that one of your child properties refers back to the parent and JSON serialization causes a circular loop.

To fix, replace this:

return Json(person, JsonRequestBehavior.AllowGet); 

with this:

return Content(JsonConvert.SerializeObject(person,                 new JsonSerializerSettings                 {                     ReferenceLoopHandling = ReferenceLoopHandling.Ignore                 }), "application/json"); 

You will have to install NewtonSoft.Json:

using Newtonsoft.Json; 

Answers 3

A circular reference was detected while serializing an object of type occurred because JSON serializer doesn't support circular references inside your object hierarchy (i.e. passing PersonnelDetailsVm that contains references to data models). To resolve this issue, use JSON.NET's JsonConvert.SerializeObject() with default settings set like this:

JsonConvert.DefaultSettings = () => new JsonSerializerSettings {     PreserveReferencesHandling = PreserveReferencesHandling.All,     ReferenceLoopHandling = ReferenceLoopHandling.Ignore }; 

Afterwards, you can return JsonResult from viewmodel:

string jsonStr = JsonConvert.SerializeObject(person);  return Json(jsonStr); 

If you're using IE and encountering save dialog because of friendly JSON errors configuration, you may need to add text/html or text/plain when returning JSON data:

return Json(jsonStr, "text/html"); 

Or hide the Json() method inside controller class like this:

protected new JsonResult Json(object data) {     if (!Request.AcceptTypes.Contains("application/json"))         return base.Json(data, "text/plain");     else         return base.Json(data); } 

Additionally, instead of return View(person); you may consider return PartialView("Details", person); because AJAX call intended to stay on same page.

Answers 4

Setting ReferenceLoopHandling = ReferenceLoopHandling.Ignore will handle your issue related to circular reference exception.

Now to about issue(s) you presented in your updates, I think you might have some misunderstanding about how Ajax request work. I think this because when you make an ajax request to server it 'll respond with JSON data which will be representation of your view model and will be agnostic to your view (cshtml) code, so when you call $("body").html(response); you are replacing content of your page with stringified representation of your JSON view model. Take away here is that when you make an ajax request only back-end code will get executed and your view code(cshtml) will not be executed, period.

To solve your you will have to replace content of the table itself and not the body of the page, so you Ajax success callback should be something like:

var tempData = {  "IBM": "IBM",  "UserName": "UserName",  "Active": false,  "CurrentWeekDates": [],   "WeighAssociations": [],  "ArrestAssociations": [],  "InspectionAssociations": [],  "WeighLocations": [],  "ArrestTypes": [],  "InspectionLevels": []  };    function onSuccess(response){  	var table = $("#tblData");    table.html("");        // code to create your head same code as your cshtml  	table.append("<thead><th>New Column</th></thead>");    table.append("<tr><td>Column 1</td></tr>");        $("#btnLoad").text("Loaded");  }    $("#btnLoad").click(function(){    $("#btnLoad").attr("disabled", "");    $("#btnLoad").text("Loading...");    // call onSuccess function after 5 second to replicate server call    setTimeout(function(){ onSuccess(tempData) }, 5000);  });

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>    <button id="btnLoad">Load Data</button>  <table id="tblData" class="table table-bordered">      <thead>          <tr>              <th></th>              <th class="text-center">Mon<br /> 01-01-1990</th>              <th class="text-center table-success">Total For Week</th>          </tr>      </thead>      <tbody>        <tr class="text-center">          <td class="table-dark">Column 1</td>          <td>Columns 2</td>          <td>0</td>          <td class="table-success font-weight-bold">0</td>        </tr>      </tbody>  </table>

I hope this helps you!

Answers 5

You have 2 different return methods and you're setting the body content to the response of the request, which if the else statement is run will be JSON and not html.

I would suggest to either always return JSON or always return a View/Partial.

else { var paramDate = DateTime.ParseExact(detailsDate, "MM/dd/yyyy", CultureInfo.CurrentCulture);  var startOfWeek = paramDate.AddDays((int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek -                                          (int)paramDate.DayOfWeek); person.CurrentWeekDates = Enumerable.Range(0, 7).Select(i => startOfWeek.AddDays(i)).ToList(); var teuFormIds = db.tbl_TEUForm     .Where(x => person.CurrentWeekDates.Contains(x.EventDate) && x.PersonnelIBM == person.IBM).Select(t => t.Id).ToList();  person.WeighAssociations = db.tbl_WeighAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList(); person.ArrestAssociations = db.tbl_TEUArrestAssc.Where(x => teuFormIds.Contains(x.TEUId)).ToList(); person.InspectionAssociations =     db.tblTEUInspectionAsscs.Where(x => teuFormIds.Contains(x.TEUId)).ToList();  JsonConvert.DefaultSettings = () => new JsonSerializerSettings() {     PreserveReferencesHandling = PreserveReferencesHandling.All,     ReferenceLoopHandling = ReferenceLoopHandling.Ignore };  var jsonStr = JsonConvert.SerializeObject(person);  //return Json(jsonStr, "text/plain"); return Partial(person); } 

Answers 6

I would suggest the below steps

1. One partial view for filters which include DateTime and Submit button(FilterPartial)
2. One Partial view for tables to be rendered(ResultsPartial)

When the body is loaded then load the first partial view(let's call FilterPartial) which will set the values in the ResultsPartial view.

function GetData(params) { $.ajax({     type: "POST",     url: '/controller/Action',     data: {         //Parameters     },     dataType: 'html',     success: function (data) {         //You can either load the html directly or render the control here      } }); 

}

  public PartialViewResult Action()     {        return PartialView("");     } 

The ViewModel logic can be kept as it is, you have to segregate the views. Let me know if this helps...

Read More

Position in JSON string to path in the object

I have a JSON string similar to this one:

{     "Version": "XXX",     "Statements": [         {...},         {...},         {...}     ] } 

How can I find out which object inside Statements property is defined at character XX of the JSON string? (considering that those objects can have arbitrarily deep nesting).

For example, if I have a string

{"Version":"XXX","Statements":[{"a":1},{"b":2},{"b":3}]} -------------------------------------------------------- 123456789 123456789 123456789 123456789 123456789 123456 

then character at position 36 would correspond to the first statement object, while character at position 52 would correspond to the third statement object.

3 Answers

Answers 1

Here is some dirty solution that doesn't require external libs:

const data = '{"Version":"XXX","Statements":[{"a":1},{"b":2},{"b":3}],"some":0}';    const getValuesPositionInArray = arrayKey => data => {    const arrayNameSeparator = `"${arrayKey}":`;    const targetArrayIndexOf = data.indexOf(arrayNameSeparator) + arrayNameSeparator.length;    const arrayStringWithRest = data.slice(targetArrayIndexOf, data.length);        const { result } = arrayStringWithRest.split('').reduce(      (acc, char, idx, array) => {        if (acc.finished) return acc;        if (!acc.processingKey && char === '[') acc.nesting += 1;        if (!acc.processingKey && char === ']') acc.nesting -= 1;                const shouldFinish = acc.nesting === 0;        const charIsDblQuote = char === '"';        const charBefore = array[idx - 1];        const charAfter = array[idx + 1];                acc.position += 1;        acc.finished = shouldFinish;          if (acc.processingKey && !charIsDblQuote) acc.processedKey += char;        if (charIsDblQuote) acc.processingKey = !acc.processingKey;        if (charIsDblQuote && !acc.processingKey && charAfter === ':') {        	acc.result[acc.processedKey] = acc.position;          acc.processedKey = '';        }                return acc;      },       {         finished: false,         processingKey: false,        processedKey: '',        nesting: 0,        position: targetArrayIndexOf + 1,        result: {}      }    )        return result;      }    const result = getValuesPositionInArray('Statements')(data);    console.log(result)

But this snippet will break if target objects would contain string values.

EDIT

And below is updated snippet with string values fix and with parsed values as well:

const data = '{"Version":"XXX","Statements":[{"aa":"some"},{"b":"ano:,{ther}"},{"bb":3}],"some":0}';    const getValuesPositionInArray = arrayKey => data => {    const arrayNameSeparator = `"${arrayKey}":`;    const targetArrayIndexOf = data.indexOf(arrayNameSeparator) + arrayNameSeparator.length;    const arrayStringWithRest = data.slice(targetArrayIndexOf, data.length);    const charsAfterValue = ['}', ','];    const charsBeforeKey = ['{', ','];      const { result } = arrayStringWithRest.split('').reduce(      (acc, char, idx, array) => {        if (acc.finished) return acc;        if (!acc.processingKey && !acc.processingValue && char === '[') acc.nesting += 1;        if (!acc.processingKey && !acc.processingValue && char === ']') acc.nesting -= 1;          const shouldFinish = acc.nesting === 0;        const charIsDblQuote = char === '"';        const charBefore = array[idx - 1];        const charAfter = array[idx + 1];               const keyProcessingStarted = (          charIsDblQuote &&          !acc.processingKey &&          !acc.processingValue &&          charsBeforeKey.includes(charBefore)        );          const keyProcessingFinished = (          charAfter === ':' &&          charIsDblQuote &&           acc.processingKey         );          const valueProcessingStarted = (          char === ':' &&          !acc.processingKey &&          !acc.processingValue        );          const valueProcessingFinished = (          (acc.lastProcessedValueType === String            ? charIsDblQuote            : true          ) &&          acc.processingValue &&          charsAfterValue.includes(charAfter)        );          acc.position += 1;        acc.finished = shouldFinish;          if (acc.processingKey && !charIsDblQuote) acc.processedKey += char;        if (acc.processingValue && !charIsDblQuote) acc.processedValue += char;                if (keyProcessingStarted) {          acc.processingKey = true;        } else if (keyProcessingFinished) {          acc.processingKey = false;          acc.result[acc.processedKey] = { position: acc.position };          acc.lastProcessedKey = acc.processedKey;          acc.processedKey = '';        }          if (valueProcessingStarted) {          acc.processingValue = true;          acc.lastProcessedValueType = charAfter === '"' ? String : Number;        } else if (valueProcessingFinished) {          acc.processingValue = false;        	acc.result[acc.lastProcessedKey].value = (            acc.lastProcessedValueType(acc.processedValue)          );          acc.processedValue = '';          acc.lastProcessedKey = '';          acc.lastProcessedValueType = (v) => v;        }          return acc;      },      {        finished: false,        processingKey: false,        processingValue: false,        processedKey: '',        processedValue: '',        lastProcessedKey: '',        lastProcessedValueType: (v) => v,        nesting: 0,        position: targetArrayIndexOf + 1,        result: {}      }    )      return result;    }    const result = getValuesPositionInArray('Statements')(data);    console.log(result)

Answers 2

After doing a bunch of research, I think I have a way forward without writing my own parser by using esprima package. Since esprima it's not JSON specific (but rather JavaScript), I have to wrap my JSON string into brackets.

Each element in tree contains loc property with a range matching it to position in original JSON string.

var esprima = require("esprima");  var JSONPath = require('JSONPath');    function getStatementIndex(str, line, column) {  	var tree = esprima.parseScript(str, {loc:true});  	var query = "$.body[0].expression.properties[?(@.key.value=='Statement')].value.elements[*].loc";  	var locations = JSONPath({json: tree, path: query});  	  	console.log(locations);  	  	for(var i = 0; i < locations.length; i++) {  		var loc = locations[i];  		  		var contains = false;  		  		if (loc.start.line < line && loc.end.line > line) {  			continue;  		}  		  		// If a single line and in between  		if (loc.start.line == loc.end.line && loc.start.line == line) {  			if (loc.start.column <= column && loc.end.column >= column) {  				contains = true;  			}  			  		// If on the beginning line  		} else if (loc.start.line == line && loc.start.column <= column) {  			contains = true;  		  		// If on the end line  		} else if (loc.end.line == line && loc.end.column >= column) {  			contains = true;  		  		// If in between  		} else if (loc.start.line < line  && loc.end.line > line) {  			contains = true;  		}  					  		if (contains)  			return i;  	}  	  	return -1;  }    var result = getStatementIndex(str, 81, 7);

Answers 3

To find the position of something in the json string, if you want to build your own algorithm, there are several things to take into account, one issue is that several strings could lead to the same object literal, also the order of properties in the objects is not guaranteed, then same string could lead to different order in the properties. We know that every . means { in the string, but [ could mean [ or { . So to find the position of 1 for example, we should remove the spaces in the original string and perform recursive loops and build a json again and find the match. Here just an example to find the position of 1:

var json = '{"Version":"XXX","Statements":[{"a":1},{"b":2},{"b":3}]}';    var obj = JSON.parse(json)    var str2 = ""    for(p in obj){      str2 += "{";      str2 += p+":";      if(p == "Statements"){          str2 += ":["          obj[p].forEach(o=>{                          for(p2 in o){                  if(p2 == "a"){                      str2 += '{"a":'                  }              }          })      }else{          str2 +='"'+obj[p]+'",'      }      }  console.log(str2)  console.log(str2.length+1)

This example is not accurate, it's just to show you one possible approach. In a real and general solution you should take into account hundreds of things.

Read More

Is it possible to change a page webmethod response in JSON, removing the .d node?

I know I can achieve this with a ASHX handler, but i really need it from a page webmethod.

I'm calling a page webmethod from in page javascript using this code:

        $.ajax({ type: 'POST', url: '<%= ResolveUrl("~/teste-datatables.aspx/getdata") %>',data: '{ a: ' + id + '}', contentType: 'application/json; charset=utf-8', dataType: 'json',                 success: function (response) {                         console.log( response);                     },                     error: function(XMLHttpRequest, textStatus, errorThrown) {                         console.log('textStatus:' + textStatus);                         console.log('errorThrown:' + errorThrown);                         console.log(XMLHttpRequest);                     }             }); 

I have a JSON file teste.json (just to keep put database interactions and facilitate tests from you guys):

{   "draw": 1,   "recordsTotal": 57,   "recordsFiltered": 57,   "data": [     [       "Airi",       "Satou",       "Accountant",       "Tokyo",       "28th Nov 08",       "$162,700"     ],     [       "Angelica",       "Ramos",       "Chief Executive Officer (CEO)",       "London",       "9th Oct 09",       "$1,200,000"     ],     [       "Ashton",       "Cox",       "Junior Technical Author",       "San Francisco",       "12th Jan 09",       "$86,000"     ],     [       "Bradley",       "Greer",       "Software Engineer",       "London",       "13th Oct 12",       "$132,000"     ],     [       "Brenden",       "Wagner",       "Software Engineer",       "San Francisco",       "7th Jun 11",       "$206,850"     ],     [       "Brielle",       "Williamson",       "Integration Specialist",       "New York",       "2nd Dec 12",       "$372,000"     ],     [       "Bruno",       "Nash",       "Software Engineer",       "London",       "3rd May 11",       "$163,500"     ],     [       "Caesar",       "Vance",       "Pre-Sales Support",       "New York",       "12th Dec 11",       "$106,450"     ],     [       "Cara",       "Stevens",       "Sales Assistant",       "New York",       "6th Dec 11",       "$145,600"     ],     [       "Cedric",       "Kelly",       "Senior Javascript Developer",       "Edinburgh",       "29th Mar 12",       "$433,060"     ]   ] } 

And tried several variations in webmethod...

Test1 Load JSON from file to string and then return it

<System.Web.Script.Services.ScriptMethod(ResponseFormat:=ResponseFormat.Json), System.Web.Services.WebMethod(True)> Public Shared Function getdata(a As Integer) As String     Dim path As String = HttpContext.Current.Server.MapPath("/teste.json")     Dim jsonString As String = File.ReadAllText(path)     Return jsonString.Replace(vbCr, "").Replace(vbLf, "") End Function 

The result is:

{d: "{  "draw": 1,  "recordsTotal": 57,  "recordsFilter ... gh",      "29th Mar 12",      "$433,060"    ]  ]}"} 

Test2 Load JSON from file to string, convert it to datablesnet object and then serealize it and write to current context

<System.Web.Script.Services.ScriptMethod(ResponseFormat:=ResponseFormat.Json), System.Web.Services.WebMethod(True)> Public Shared Sub getdata6(a As Integer)     Dim path As String = HttpContext.Current.Server.MapPath("/teste.json")     Dim jsonString As String = File.ReadAllText(path)     Dim mytable As datatablesnet = Newtonsoft.Json.JsonConvert.DeserializeObject(Of datatablesnet)(jsonString)     Dim serializer As New JavaScriptSerializer     HttpContext.Current.Response.ContentType = "application/json"     HttpContext.Current.Response.Write(serializer.Serialize(mytable)) End Sub 

I get an error and analising the response i see that it was (was added in the end {"d":null}:

{"draw":1,"recordsTotal":57,"recordsFiltered":57,"data":[["Airi","Satou","Accountant","Tokyo","28th Nov 08","$162,700"],["Angelica","Ramos","Chief Executive Officer (CEO)","London","9th Oct 09","$1,200,000"],["Ashton","Cox","Junior Technical Author","San Francisco","12th Jan 09","$86,000"],["Bradley","Greer","Software Engineer","London","13th Oct 12","$132,000"],["Brenden","Wagner","Software Engineer","San Francisco","7th Jun 11","$206,850"],["Brielle","Williamson","Integration Specialist","New York","2nd Dec 12","$372,000"],["Bruno","Nash","Software Engineer","London","3rd May 11","$163,500"],["Caesar","Vance","Pre-Sales Support","New York","12th Dec 11","$106,450"],["Cara","Stevens","Sales Assistant","New York","6th Dec 11","$145,600"],["Cedric","Kelly","Senior Javascript Developer","Edinburgh","29th Mar 12","$433,060"]]}{"d":null} 

Test3 Load JSON from file to string, convert it to datablesnet object and then serealize it and return it

<System.Web.Script.Services.ScriptMethod(ResponseFormat:=ResponseFormat.Json), System.Web.Services.WebMethod(True)> Public Shared Function getdata5(a As Integer) As String     Dim path As String = HttpContext.Current.Server.MapPath("/teste.json")     Dim jsonString As String = File.ReadAllText(path)     Dim mytable As datatablesnet = Newtonsoft.Json.JsonConvert.DeserializeObject(Of datatablesnet)(jsonString)     Dim serializer As New JavaScriptSerializer      Return serializer.Serialize(mytable) End Function 

The result is the same from Test1:

{d: "{  "draw": 1,  "recordsTotal": 57,  "recordsFilter ... gh",      "29th Mar 12",      "$433,060"    ]  ]}"} 

I really need to get rid of the .d node, receiving the JSON directly from response and not from response.d. Is there any way I can achieve this from page webmethod ?

1 Answers

Answers 1

I really need to get rid of the .d node, receiving the JSON directly from response and not from response.d. Is there any way I can achieve this from page WebMethod?

Yes, there is. You can write the result directly to response:

<System.Web.Services.WebMethod(True)> Public Shared Sub getdata(a As Integer)     Dim path As String = HttpContext.Current.Server.MapPath("/teste.json")     Dim jsonString As String = System.IO.File.ReadAllText(path)     HttpContext.Current.Response.Clear()     HttpContext.Current.Response.ContentType = "application/json; charset=utf-8"     HttpContext.Current.Response.Write(jsonString)     HttpContext.Current.Response.Flush()     HttpContext.Current.Response.End() End Sub 

In case that you need to do it in multiple classes in your code, you can easily create a shared Sub WriteJsonToResponse(obj as Object) and in the body of the method, first serialize the obj to json using JavaScriptSerializer or Newtonsoft.JsonConvert, then write it to the response like above method.

Read More

Custom Jackson serializer for a generic tree

Say I have a parametrized tree implemented in Java as follows:

public class Tree<E> {    private static class Node {       E element;       List<Node> children.    }     Node root;     //... You get the idea. } 

The idea here is that the implementation above is only concerned with the topology of the tree, but does not know anything about the elements that will be stored in the tree by an instantiation.

Now, say I want my tree elements to be geographies. The reason they are organized in trees is because continents contain countries, countries contain states or a provinces, and so on. For simplicity, a geography has a name and a type:

public class GeoElement { String name; String type; } 

So that, finally, the geo hierarchy looks like so:

public class Geography extends Tree<GeoElement> {} 

Now to Jackson serialization. Assuming the Jackson serializer can see the fields, the direct serialization of this implementation will look like this:

{    "root": {       "element": {          "name":"Latin America",          "type":"Continent"       }       "children": [           {              "element": {                 "name":"Brazil",                 "type":"Country"              },              "children": [                  // ... A list of states in Brazil              ]           },           {              "element": {                 "name":"Argentina",                 "type":"Country"              },              "children": [                  // ... A list of states in Argentina              ]           }       ]    } 

This JSON rendering is no good because it contains the unnecessary artifacts from the Tree and Node classes, i.e. "root" and "element". What I need instead is this:

{    "name":"Latin America",    "type":"Continent"    "children": [        {           "name":"Brazil",           "type":"Country"           "children": [              // ... A list of states in Brazil           ]        },        {           "name":"Argentina",           "type":"Country"           "children": [              // ... A list of states in Argentina           ]        }    ] } 

Any help is most appreciated. -Igor.

4 Answers

Answers 1

What you need is @JsonUnwrapped.

Annotation used to indicate that a property should be serialized "unwrapped"; that is, if it would be serialized as JSON Object, its properties are instead included as properties of its containing Object

Add this annotation to the root field of Tree & element field of Node classes as follows:

public class Tree<E> {    private static class Node {        @JsonUnwrapped       E element;       List<Node> children.    }     @JsonUnwrapped    Node root;     //... You get the idea. } 

And it will give you your desired output:

{     "name": "Latin America",     "type": "Continent",     "children": [{         "name": "Brazil",         "type": "Country",         "children": []     }, {         "name": "Argentina",         "type": "Country",         "children": []     }] } 

Answers 2

Perhaps use @JsonValue like so:

public class Tree<E> {   @JsonValue   Node root; } 

if all you need is to just "unwrap" your tree?

Answers 3

Your best bet will be to build and register a custom serializer for your objects.

Define your serializer:

public class NodeSerializer extends StdSerializer<Node> { 

Then on your Node class:

@JsonSerialize(using = NodeSerializer.class) public class Node { } 

And inside of the NodeSerializer

@Override     public void serialize(       Node node, JsonGenerator jgen, SerializerProvider provider)        throws IOException, JsonProcessingException {          jgen.writeStartObject();         jgen.writeStringField("name", node.element.name);          jgen.writeStringField("type", node.element.type);         //Process children         serializeFields(node, jgen, provider);         jgen.writeEndObject();     } 

This general framework will let you control how the elements get serialized. You may need to @JsonIgnore the element object inside of the Node as well since your custom serializer is taking care of pushing that info into the resulting JSON. There is a lot online about custom serializers and overriding default JSON export.

You can get rid of root in a similar way with a serializer for the Tree implementation.

If you don't want to register the serializer on the class you can also do it on a one at a time basis using the ObjectMapper:

ObjectMapper mapper = new ObjectMapper();  SimpleModule module = new SimpleModule(); module.addSerializer(Node.class, new NodeSerializer()); mapper.registerModule(module);  String serialized = mapper.writeValueAsString(tree); 

The annotation approach will apply globally. This approach allows some control of how/where your custom serializer is used.

Answers 4

For removing the element type, one possibility would be to change your structure so that the name and the type will be directly included in each node:

public class TreeGeo {    private static class Node {       String name;       String type;       List<Node> children.    }     Node root; } 

For removing the root type, I don't know. I suppose that you could extract a sub-object from the jsonObject but I don't know much about Jackson. However, you could give it a better name like world or manipulate the resulting string to remove it manually with some string manipulations.

Read More

Valid JSON Load in Python file

Running into a problem with my JSON:

First issue was that SyntaxError: Non-ASCII character '\xe2' in file so I added # -*- coding: utf-8 -*- at the top of my file.

Then the problem became a problem where I load my JSON x = json.loads(x): ValueError: Expecting , delimiter: line 3 column 52 (char 57). I referenced this stackoverflow solution and so added an r in front of my JSON:

x = r"""[   { my validated json... } ]""" 

But then I get an error TypeError: sequence item 3: expected string or Unicode, NoneType found - I think it that the r is throwing it off somehow?

JSON Resembles the following:

[   {     "brief": "Brief 1",     "description": "Description 1",     "photos": [       "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example.jpg?0101010101010",       "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example2.jpg?0101010101010",       "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example3.jpg?0101010101010"     ],     "price": "145",     "tags": [       "tag1",       "tag2",       "tag3"     ],     "title": "Title 1"   },   {     "brief": "Brief 2",     "description": "Description 2",     "photos": [       "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example4.jpg?0101010101010",       "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example5.jpg?0101010101010"     ],     "price": "150",     "tags": [       "tag4",       "tag5",       "tag6",       "tag7",       "tag8"     ],     "title": "Title 2"   },{     "brief": "blah blah 5'0\" to 5'4\"",     "buyerPickup": true,     "condition": "Good",     "coverShipping": false,         "description": "blah blah 5'0\" to 5'4\". blah blah.Size L/20”\n 5’8-5’11\n29lbs\n3x7 speed\n\n  \r\n\r\n",     "photos": [       "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-010101.jpeg?11111",       "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-020202?111111"     ],     "price": "240",     "tags": [       "tag2",       "5'0\"-5'4\""     ],     "title": "blah blah 17\" Frame",     "front": "https://firebasestorage.googleapis.com/v0/b/example.appspot.com/o/Images%2F0007891113.jpg?alt=media&token=111-11-11-11-111"       }  ] 

CURRENT CODE

# -*- coding: utf-8 -*-  import csv import json  x = """[       {         "brief": "Brief 1",         "description": "Description 1",         "photos": [           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example.jpg?0101010101010",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example2.jpg?0101010101010",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example3.jpg?0101010101010"         ],         "price": "145",         "tags": [           "tag1",           "tag2",           "tag3"         ],         "title": "Title 1"       },       {         "brief": "Brief 2",         "description": "Description 2",         "photos": [           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example4.jpg?0101010101010",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example5.jpg?0101010101010"         ],         "price": "150",         "tags": [           "tag4",           "tag5",           "tag6",           "tag7",           "tag8"         ],         "title": "Title 2"       },{         "brief": "blah blah 5'0\" to 5'4\"",         "buyerPickup": true,         "condition": "Good",         "coverShipping": false,             "description": "blah blah 5'0\" to 5'4\". blah blah.Size L/20”\n 5’8-5’11\n29lbs\n3x7 speed\n\n  \r\n\r\n",         "photos": [           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-010101.jpeg?11111",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-020202?111111"         ],         "price": "240",         "tags": [           "tag2",           "5'0\"-5'4\""         ],         "title": "blah blah 17\" Frame",         "front": "https://firebasestorage.googleapis.com/v0/b/example.appspot.com/o/Images%2F0007891113.jpg?alt=media&token=111-11-11-11-111"           }      ]"""  x = json.loads(x)  f = csv.writer(open("example.csv", "wb+"))  f.writerow(["Handle","Title","Body (HTML)", "Vendor","Type","Tags","Published","Option1 Name","Option1 Value","Variant Inventory Qty","Variant Inventory Policy","Variant Fulfillment Service","Variant Price","Variant Requires Shipping","Variant Taxable","Image Src"])      for x in x:          allTags = "\"" + ','.join(x["tags"]) + "\""         images = x["photos"]         f.writerow([x["title"],                     x["title"],                     x["description"],                     "Vendor Name",                     "Widget",                     allTags,                     "TRUE",                     "Title",                     "Default Title",                     "1",                     "deny",                     "manual",                     x["price"],                     "TRUE",                     "TRUE",                     images.pop(0) if images else None])         while images:             f.writerow([x["title"],None,None,None,None,None,None,None,None,None,None,None,None,None,None,images.pop(0)]) 

ERROR MESSAGE: Full traceback that I see: Traceback (most recent call last):

Traceback (most recent call last): File "runnit2.py", line 976, in <module> allTags = "\"" + ','.join(x["tags"]) + "\"" TypeError: sequence item 3: expected string or Unicode, NoneType found

UPDATE: I've identified that the data, specifically [x["title"], x["title"],x["description"], has some characters that the code doesn't like. 'ascii' codec can't encode character u'\u201d' in position 9: ordinal not in range(128). I've done a quick fix with x["description"].encode('utf-8'), etc., but it pretty much eliminates everything that's in that cell. Is there is a better way which doesn't delete everything after offending character?

4 Answers

Answers 1

Use raw string and set file encoding to utf-8 in normal (non-binary mode) mode when opening. For Python 3.6 it will be enough.

On Python 2.7 you should use codecs.open('example.csv', 'w', encoding='utf-8') instead of regular open() when dealing with unicode content. Also, csv module on Python 2.7 does not support unicode out of the box, so I suggest switching to unicodecsv or following the guidelines in this answer.

Answers 2

Modify reading and writing using W If you must use WB, use the following functions. You need to add r in front of all texts to handle special symbols.

import csv import json  x = r"""[       {         "brief": "Brief 1",         "description": "Description 1",         "photos": [           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example.jpg?0101010101010",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example2.jpg?0101010101010",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example3.jpg?0101010101010"         ],         "price": "145",         "tags": [           "tag1",           "tag2",           "tag3"         ],         "title": "Title 1"       },       {         "brief": "Brief 2",         "description": "Description 2",         "photos": [           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example4.jpg?0101010101010",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-example5.jpg?0101010101010"         ],         "price": "150",         "tags": [           "tag4",           "tag5",           "tag6",           "tag7",           "tag8"         ],         "title": "Title 2"       },{         "brief": "blah blah 5'0\" to 5'4\"",         "buyerPickup": true,         "condition": "Good",         "coverShipping": false,             "description": "blah blah 5'0\" to 5'4\". blah blah.Size L/20”\n 5’8-5’11\n29lbs\n3x7 speed\n\n  \r\n\r\n",         "photos": [           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-010101.jpeg?11111",           "https://cdn.shopify.com/s/files/1/01/01/01/files/imgs-020202?111111"         ],         "price": "240",         "tags": [           "tag2",           "5'0\"-5'4\""         ],         "title": "blah blah 17\" Frame",         "front": "https://firebasestorage.googleapis.com/v0/b/example.appspot.com/o/Images%2F0007891113.jpg?alt=media&token=111-11-11-11-111"           }      ]"""  x = json.loads(x)   def to_str(bytes_or_str):     if isinstance(bytes_or_str, bytes):         value = bytes_or_str.encode('utf-8')     else:         value = bytes_or_str     return value   def to_bytes(bytes_or_str):     if isinstance(bytes_or_str, str):         value = bytes_or_str.encode('utf-8')     else:         value = bytes_or_str      return value   f = csv.writer(open("example.csv", "w+")) writeList = ["Handle", "Title", "Body (HTML)", "Vendor", "Type", "Tags", "Published", "Option1 Name", "Option1 Value",              "Variant Inventory Qty", "Variant Inventory Policy", "Variant Fulfillment Service", "Variant Price",              "Variant Requires Shipping", "Variant Taxable", "Image Src"] newList = [] for item in writeList:     newList.append(to_bytes(item))  f.writerow(newList)  for x in x:      allTags = r"\"" + ','.join(x["tags"]) + r"\""     images = x["photos"]     f.writerow([x["title"],                 x["title"],                 x["description"],                 "Vendor Name",                 "Widget",                 allTags,                 "TRUE",                 "Title",                 "Default Title",                 "1",                 "deny",                 "manual",                 x["price"],                 "TRUE",                 "TRUE",                 images.pop(0) if images else None])     while images:         f.writerow([x["title"], None, None, None, None, None, None, None, None, None, None, None, None, None, None,                     images.pop(0)]) 

Answers 3

From your posted sample data, I assume that the 1st index of the posted json has a null in the 3rd index of the values of tag key. i.e: tag7

"tags": [           "tag4",           "tag5",           "tag6",           "tag7",           "tag8"         ], 

To get rid of the TypeError that raises due to nulls you can simply check and replace the nulls if they exist as shown below.

x["tags"] = ["" if i is None else i for i in x["tags"]] allTags = "\"" + ','.join(x["tags"]) + "\"" 

I have assigned an empty string to replace nulls.

Alternatively you can remove all the false elements by using None in the filter() function.

allTags = "\"" + ','.join(filter(None, x["tags"])) + "\"" 

NOTE: Add r"[...]" and fix the indentation issue in the for loop.

Answers 4

Possible duplicate of this question how to convert characters like \x22 into string

On cleaning the code the error boils down to

import json  x = '''   {     "brief": "\""   }'''  x = json.loads(x) 

Consider replacing \" with \u201d

import json  x = '{"brief": "\u201d"}'  x = json.loads(x) 

Read More

How to avoid generating crc files and SUCCESS files while saving a DataFrame?

I am using the following code to save a spark DataFrame to JSON file

unzipJSON.write.mode("append").json("/home/eranw/Workspace/JSON/output/unCompressedJson.json") 

the output result is:

part-r-00000-704b5725-15ea-4705-b347-285a4b0e7fd8 .part-r-00000-704b5725-15ea-4705-b347-285a4b0e7fd8.crc part-r-00001-704b5725-15ea-4705-b347-285a4b0e7fd8 .part-r-00001-704b5725-15ea-4705-b347-285a4b0e7fd8.crc _SUCCESS ._SUCCESS.crc 

1. How do I generate a single JSON file and not a file per line?
2. How can I avoid the *crc files?
3. How can I avoid the SUCCESS file?

1 Answers

Answers 1

If you want a single file, you need to do a coalesce to a single partition before calling write, so:

unzipJSON.coalesce(1).write.mode("append").json("/home/eranw/Workspace/JSON/output/unCompressedJson.json") 

Personally, I find it rather annoying that the number of output files depend on number of partitions you have before calling write - especially if you do a write with a partitionBy - but as far as I know, there are currently no other way.

I don't know if there is a way to disable the .crc files - I don't know of one - but you can disable the _SUCCESS file by setting the following on the hadoop configuration of the Spark context.

sc.hadoopConfiguration.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false") 

Note, that you may also want to disable generation of the metadata files with:

sc.hadoopConfiguration.set("parquet.enable.summary-metadata", "false") 

Apparently, generating the metadata files takes some time (see this blog post) but aren't actually that important (according to this). Personally, I always disable them and I have had no issues.

Read More

User/Role/Permission for application using MongoDB and Jackson JSON?

I have created three classes user, role, permission. I'm creating an application which is using mongodb and Jackson JSON (wiki.fasterxml.com/JacksonHome)

Example,

 User will have Role Object  Role will have List<Permission> and List<User> Objects  Permission will have name and description field 

In mongoose, we have DBRef/ref attribute to specify parent, child relationship so that the child entities will be automatically populated on query using populate() api

Questions:

1. Do we have annotation in fasterxml similar to DBRef/Ref in mongoose?
2. Is there a way to retrieve all the child objects of the parent in single query?

0 Answers

Read More

Meteor Http Client API Call Gzip uncompress resultant JSON not working

I don't know if it's the exact issue, but from what I can understand it is. I am making an external API call with the HTTP.call and trying to parse the response as JSON (which it is), but I'm getting strange content returned, which I think is Gzipped content. I'm setting the npmRequestOptions parameter for gzip to true, but it is still returning the same content.

Here's my code:

var result; try {   result = HTTP.call('GET', 'http://atlas.atdw-online.com.au/api/atlas/products', {     params: {       key: '594c18ea1a184997a5457593b5873c8f',       att: 'INDIGENOUSAUTHARTS',       out: 'json',     },     npmRequestOptions : {gzip : true, json: true},     headers: {       // "Accept": "application/json",       'Content-Type' : 'application/json; charset=UTF-16',       "OSLC-Core-Version": "2.0",     },    });  } catch (e) {   // Got a network error, timeout, or HTTP error in the 400 or 500 range. }  // Append data in correct experiences format. var content = JSON.parse(result.content); 

The error is as per the screenshot:

Here are the response headers from the API call which seems to indicate the content is being gzipped:

HTTP/1.1 200 OK Access-Control-Allow-Credentials: true CF-RAY: 4291c15b957d4d88-PER Content-Encoding: gzip Content-Type: application/json; charset=utf-16 Date: Mon, 11 Jun 2018 05:42:36 GMT ETag: W/"863e-ylc8HuIAiQAynP7anYY1GA" Server: cloudflare Vary: Origin, Accept-Encoding X-Powered-By: Express 

I believe npmRequestOptions : {gzip : true, json: true} should be working, but it's not. There's no additional information available and I've been searching all day.

1 Answers

Answers 1

So I've figured out the solution to this was not in the Meteor app itself, it was in the content being returned. The response is UTF-16LE so I had to remove special characters from the response to I could JSONify it.

result = result.content.replace(/[^A-Za-z0-9{}@&:\-_() /\\"=[\],.?!]/g, '') 

Read More

Mysql json based trending tags implementation

I am trying to identifying the trending tags (based on maximum hits) on time series using mysql json feature. Below is my table

CREATE TABLE TAG_COUNTER (     account       varchar(36) NOT NULL,     time_id       INT NOT NULL,     counters      JSON,     PRIMARY KEY   (account, time_id) ) 

In every web api request, i will be getting the multiple different tags per account, and based on number of tags, i will prepare the INSERT ON DUPLICATE KEY UPDATE query. Below example is showing insertion with two tags.

INSERT INTO `TAG_COUNTER`   (`account`, `time_id`, `counters`) VALUES   ('google', '2018061023', '{"tag1": 1, "tag2": 1}') ON DUPLICATE KEY UPDATE `counters` =   JSON_SET(`counters`,            '$."tag1"',            IFNULL(JSON_EXTRACT(`counters`,                         '$."tag1"'), 0) + 1,            '$."tag2"',            IFNULL(JSON_EXTRACT(`counters`,                         '$."tag2"'), 0) + 1   ); 

time_id is yyyyMMddhh, and it is hourly aggregation on each row.

Now my problem is retrival of treding tags. Below query will give me aggregation for tag1, but we will not be knowing the tags before making this query.

SELECT SUBSTRING(time_id, 1, 6) AS month, SUM(counters->>'$.tag1') FROM TAG_COUNTER WHERE counters->>'$.tag1' > 0 GROUP BY month; 

So i need generic group by query along with order by to get the trending tags for the time hourly/daily/monthly.

The sample of output expected is

Time(hour/day/month)  Tag_name  Tag_count_value(total hits) 

When i have searched the web, every where it is mentioned like below {"tag_name": "tag1", "tag_count": 1} instead of direct {"tag1" : 1} and they were using tag_name in the group by.

Q1) So is it always mandatory to have common known json key to perform group by ..?

Q2) If i have to go with this way, then what is the change in my INSERT ON DUPLICATE KEY UPDATE query for this new json label/value struture? Since the counter has to be created when it is not existing and should increment by one when it is existing.

Q3) do i have to maintain array of objects

[  {"tag_name": "tag1", "tag_count": 2},  {"tag_name": "tag2", "tag_count": 3} ] 

OR object of objects like below?

{  {"tag_name": "tag1", "tag_count": 2},  {"tag_name": "tag2", "tag_count": 3} } 

So which is better above json structure interms of INSERT and RETRIEVAL of trending count?

Q4) Can i go with existing {"key" : "value"} format instead of {"key_label" : key, "value_lable" : "value"} and possible to extract trending ..? since i am thinking that {"key" : "value"} is very straight forward and good at performance wise.

Q5) While retrieving i am using SUBSTRING(time_id, 1, 6) AS month. Will it be able to use index?

OR do i need to create multiple columns like time_hour(2018061023), time_day(20180610), time_month(201806) and use query on specific columns?

OR can i use mysql date-time functions? will that use index for faster retrieval?

Please help.

2 Answers

Answers 1

I don't see a good reason, why you use JSON here. It's also not clear, why you believe that a "nosql schema" within MySQL would do anything better.

What you probably need is something like this:

CREATE TABLE TAG_COUNTER (     account       varchar(36) NOT NULL,     time_id       INT NOT NULL,     tag_name      varchar(50) NOT NULL,     counter       INT UNSIGNED NOT NULL,     PRIMARY KEY   (account, time_id, tag_name) ); 

This will simplify your queries. The INSERT statement would look like:

INSERT INTO TAG_COUNTER   (account, time_id, tag_name, counter) VALUES   ('google', 2018061023, 'tag1', 1),   ('google', 2018061023, 'tag2', 1) ON DUPLICATE KEY UPDATE counter = counter + VALUES(counter); 

The SELECT statement might be something like this

SELECT     SUBSTRING(time_id, 1, 6) AS month,     tag_name,     SUM(counter) AS counter_agg FROM TAG_COUNTER GROUP BY month, tag_name ORDER BY month, counter_agg DESC; 

Note that I did't try to optimize the table/schema for data size and performance. That would be a different question. But you must see, that the queries are much simpler now.

Answers 2

As I have said in comments, I think moving away from JSON is the way to go. However, if you want to keep using JSON, this function (a direct copy of the one in my answer to this question, see an explanation of what it does there) and procedure will do what you want.

DELIMITER // DROP FUNCTION IF EXISTS json_merge_sum // CREATE FUNCTION json_sum_merge(IN j1 JSON, IN total JSON) RETURNS JSON BEGIN   DECLARE knum INT DEFAULT 0;   DECLARE jkeys JSON DEFAULT JSON_KEYS(j1);   DECLARE kpath VARCHAR(30);   DECLARE v INT;   DECLARE l INT DEFAULT JSON_LENGTH(jkeys);   kloop: LOOP     IF knum >= l THEN       LEAVE kloop;     END IF;     SET kpath = CONCAT('$.', JSON_EXTRACT(jkeys, CONCAT('$[', knum, ']')));     SET v = JSON_EXTRACT(j1, kpath);     IF JSON_CONTAINS_PATH(total, 'one', kpath) THEN       SET total = JSON_REPLACE(total, kpath, JSON_EXTRACT(total, kpath) + v);     ELSE       SET total = JSON_SET(total, kpath, v);     END IF;     SET knum = knum + 1;   END LOOP kloop;   RETURN total; END // 

The procedure is similar to the one in my other answer, in that it finds all the distinct tags associated with a given substring of time_id (specified as a parameter) and sums the values associated with each tag. The individual tags and counts are then written to a temporary table, from which a selection is then made grouping by time period and tag name.

DELIMITER // DROP PROCEDURE IF EXISTS count_tags // CREATE PROCEDURE count_tags(IN period VARCHAR(50)) BEGIN   DECLARE finished INT DEFAULT 0;   DECLARE timeval VARCHAR(20);   DECLARE knum, l INT;   DECLARE jkeys JSON;   DECLARE time_cursor CURSOR FOR SELECT DISTINCT time_id FROM tag_counter;   DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished=1;   CREATE TEMPORARY TABLE tag_counts (Time VARCHAR(20), Tag_Name VARCHAR(30), Tag_count_value INT, INDEX(Time, Tag_Name));   OPEN time_cursor;   time_loop: LOOP     FETCH time_cursor INTO timeval;     IF finished=1 THEN       LEAVE time_loop;     END IF;     SET @total = '{}';     SET @query = CONCAT("SELECT MIN(@total:=json_sum_merge(counters, @total)) INTO @json FROM TAG_COUNTER WHERE time_id='", timeval, "'");     PREPARE stmt FROM @query;     EXECUTE stmt;     DEALLOCATE PREPARE stmt;     SET @query = CONCAT('INSERT INTO tag_counts VALUES(', period, ', ?, ?)');     PREPARE stmt FROM @query;     SET @timeval = timeval;     SET l = JSON_LENGTH(@total);     SET jkeys = JSON_KEYS(@total);     SET knum = 0;     key_loop: LOOP       IF knum >= l THEN         LEAVE key_loop;       END IF;       SET @k = JSON_EXTRACT(jkeys, CONCAT('$[', knum, ']'));       SET @t = JSON_EXTRACT(@total, CONCAT('$.', @k));       EXECUTE stmt USING @k, @t;       SET knum = knum + 1;     END LOOP key_loop;     DEALLOCATE PREPARE stmt;   END LOOP time_loop;   SELECT Time, Tag_Name, SUM(Tag_count_value) AS Tag_count_value FROM tag_counts GROUP BY Time, Tag_Name;   DROP TABLE tag_counts; END 

A couple of examples based on some limited sample data from your prior question. In these examples @timeval is equivalent to the time_id column. Input data:

account     time_id     counters google      20180510    {"gmail_page_viewed": 2, "search_page_viewed": 51} google      20180511    {"gmail_page_viewed": 3, "search_page_viewed": 102} apple       20180511    {"apple_page_viewed": 5, "search_page_viewed": 16} 

CALL count_tags('@timeval'):

Time        Tag_Name                Tag_count_value 20180510    "gmail_page_viewed"     2 20180510    "search_page_viewed"    51 20180511    "apple_page_viewed"     5 20180511    "gmail_page_viewed"     3 20180511    "search_page_viewed"    118 

CALL count_tags('SUBSTRING(@timeval, 1, 6)'):

Time    Tag_Name                Tag_count_value 201805  "apple_page_viewed"     5 201805  "gmail_page_viewed"     5 201805  "search_page_viewed"    169 

Note that you can also use json_sum_merge to simplify your INSERT query e.g.

INSERT INTO `TAG_COUNTER`   (`account`, `time_id`, `counters`) VALUES   ('apple', '20180511', '{"apple_page_viewed": 9, "itunes_page_viewed": 4}') ON DUPLICATE KEY UPDATE `counters` = json_sum_merge(VALUES(counters), counters) 

Result:

account     time_id     counters apple       20180511    {"apple_page_viewed": 14, "itunes_page_viewed": 4, "search_page_viewed": 16} 

In terms of the specific questions in your answer:

1. No. This answer shows it can be done with your existing data format.
2. Not applicable.
3. Not applicable.
4. Yes, you can stick with the existing {"key" : "value"} format
5. Since we have to go through every entry of tag_counter to get the list of tags, an index is not beneficial for that section. For the temporary table I have included indexes on the Time and Tag_Name columns which should benefit speed as they are used directly in the GROUP BY clause.

If you were to maintain a list of keys (e.g. in a separate table, maintained by a trigger on insert/update/delete to tag_counter) this code could be made a lot simpler and more efficient. But that is for another question.

Read More

Should I use optional for properties of object models that will be parsed from JSON?

My iOS app has a pretty common setup: it makes HTTP queries to an API server that responds with JSON objects. These JSON objects are then parsed to appropriate Swift objects.

Initially I divided properties into required properties and optional properties, mostly based on my API server's database requirements. For example, id, email, and name are require fields so they use non-optional types. Others can be NULL in database, so they are optional types.

class User {   let id: Int   let email: String   let profile: String?   let name: String   let motive: String?   let address: String?   let profilePhotoUrl: String? } 

Recently, I started wondering whether this was a good setup at all. I found out that although some properties might be always in the database, that does not mean that those properties will always be included in the JSON response.

For example, in the User profile page, all these fields are needed to properly display the view. Therefore, JSON response will include all these fields. For a view that lists users' names, however, I would not need email or id, and JSON response should probably not include those properties either. Unfortunately, this will cause error and crash the app when parsing JSON response into Swift object since the app expects id, email, name to be always not-nil.

I'm thinking of changing all properties of Swift objects into optionals, but it feels like throwing away all the benefits of this language-specific feature. Moreover, I will have to write many more lines of code to unwrap all these optionals somewhere else in the app anyway.

On the other hand, JSON objects are by their nature not very interoperable with strict static typing and nil-checking of Swift so it might be better to simply accept that annoyance.

Should I transition to models with every property as optionals? Or is there a better way? I'd appreciate any comment here.

10 Answers

Answers 1

There are three ways you can go with this:

1. Always send all the JSON data, and leave your properties non-optional.

2. Make all the properties optional.

3. Make all the properties non-optional, and write your own init(from:) method to assign default values to missing values, as described in this answer.

All of these should work; which one is "best" is opinion-based, and thus out of the scope of a Stack Overflow answer. Choose whichever one is most convenient for your particular need.

Answers 2

The first thing to do is ask: Does an element of the “view that lists users' names” need to be the same kind of object as the model object behind a “User profile page”? Perhaps not. Maybe you should create a model specifically for the user list:

struct UserList: Decodable {      struct Item: Decodable {         var id: Int         var name: String     }      var items: [Item]  } 

(Although the question said the JSON response might not include id, it doesn't seem like a user list without ids with be particularly useful, so I made it required here.)

If you really want them to be the same kind of object, then maybe you want to model a user as having core properties that the server always sends, and a “details” field that might be nil:

class User: Decodable {     let id: Int     let name: String     let details: Details?      struct Details: Decodable {         var email: String         var profile: String?         var motive: String?         var address: String?         var profilePhotoUrl: String?     }      init(from decoder: Decoder) throws {         let container = try decoder.container(keyedBy: CodingKeys.self)         id = try container.decode(Int.self, forKey: .id)         name = try container.decode(String.self, forKey: .name)         details = container.contains(.email) ? try Details(from: decoder) : nil     }      enum CodingKeys: String, CodingKey {         case id         case name          case email // Used to detect presence of Details     } } 

Note that I create the Details, if it's present, using Details(from: decoder), instead of the usual container.decode(Details.self, forKey: .details). I do it using Details(from: decoder) so that the properties of the Details come out of the same JSON object as the properties of the User, instead of requiring a nested object.

Answers 3

If the server is giving Null value for the other properties, you can go for optionals and safe unwrap. Or while unwrapping you can assign empty string to property if the value is nil

profile = jsonValue ?? "" 

Other case since the other properties are String data type you can assign default value as a empty string

class User {   let id: Int   let email: String   let profile: String = ""   let name: String   let motive: String = ""   let address: String = ""   let profilePhotoUrl: String = "" } 

Answers 4

I typically make all non-critical properties optional, and then have a failable initializer. This allows me to better handle any changes in the JSON format or broken API contracts.

For example:

class User {   let id: Int   let email: String   var profile: String?   var name: String?   var motive: String?   var address: String?   var profilePhotoUrl: String? } 

This means that I will never have a user object without an id or email (let's assume those are the two that always need to be associated with a user). If I get a JSON payload without an id or email, the Initializer in the User class will fail and won't create the user object. I then have error handling for failed initializers.

I'd much rather have a swift class with optional properties than a bunch of properties with an empty string value.

Answers 5

Yes, you should use optional if the property is not necessary in API and if you want some value in the mandatory property then assign blank value:

class User {   let id: Int?   let email: String? = ""   let profile: String?   let name: String? = ""   let motive: String?   let address: String?   let profilePhotoUrl: String? } 

Answers 6

In my opinion, I will choose 1 of 2 solutions:

1. Edit my init func from JSON to object, init with default object values for all props (id = -1, email = ''), then read JSON with optional checking.
2. Create a new class/struct for that specific case.

Answers 7

The premise:

Partial representing is a common pattern in REST. Does that mean all properties in Swift need to be optionals? For example, the client might just need a list of user ids for a view. Does that mean that all the other properties (name, email, etc) need to be marked as optional? Is this good practice in Swift?

Marking properties optional in a model only indicates that the key may or may not come. It allows the reader to know certain things about the model in the first look itself.
If you maintain only one common model for different API response structures and make all the properties optional, whether that's good practice or not is very debatable.
I have done this and it bites. Sometimes it's fine, sometimes it's just not clear enough.

Keeping one model for multiple APIs is like designing one ViewController with many UI elements and depending on particular cases, determining what UI element should be shown or not.
This increases the learning curve for new developers as there's more understanding-the-system involved.

---

My 2 cents on this:

Assuming we are going ahead with Swift's Codable for encoding/decoding models, I would break it up into separate models rather than maintaining a common model with all optionals &/or default values.

Reasons for my decision are:

1. Clarity of Separation

   • Each model for a specific purpose
   • Scope of cleaner custom decoders
     • Useful when the json structure needs a little pre-processing

2. Consideration of API specific additional keys that might come later on.

   • What if this User list API is the only one requiring more keys like, say, number of friends or some other statistic?
     • Should I continue to load a single model to support different cases with additional keys that come in only one API response but not another?
     • What if a 3rd API is designed to get user information but this time with a slightly different purpose? Should I over-load the same model with yet more keys?
   • With a single model, as the project continues to progress, things could get messy as key availability in now very API-case-based. With all being optionals we will have alot of optional bindings & maybe some shortcut nil coalescings here and there which we could have avoided with dedicated models in the first place.
3. Writing up a model is cheap but maintaining cases is not.

---

However, if I was lazy and I have a strong feeling crazy changes aren't coming up ahead, I would just go ahead making all the keys optionals and bear the associated costs.

Answers 8

This really depends on the way you are handling your data. If you are handling your data through a "Codable" class, then you have to write a custom decoder to throw an exception when you don't get certain expected values. Like so:

 class User: Codable {     let id: Int     let email: String     let profile: String?     let name: String     let motive: String?     let address: String?     let profilePhotoUrl: String?       //other methods (such as init, encoder, and decoder) need to be added below.     } 

Because I know that I'm going to need to return an error if I don't get the minimum required parameters, you would need something like an Error enum:

    enum UserCodableError: Error {          case missingNeededParameters          //and so on with more cases     } 

You should be using coding keys to keep things consistent from the server. A way to do that inside of the User Object would be like so:

    fileprivate enum CodingKeys: String, CodingKey {        case id = "YOUR JSON SERVER KEYS GO HERE"        case email        case profile        case name        case motive        case address        case profilePhotoUrl     } 

Then, you need to write your Decoder. A way to do that would be like so:

    required init(from decoder: Decoder) throws {     let values = try decoder.container(keyedBy: CodingKeys.self)     guard let id = try? values.decode(Int.self, forKey: .id), let email = try? values.decode(String.self, forKey: .email), let name = try? values.decode(String.self, forKey: .name) else {         throw UserCodableError.missingNeededParameters     }      self.id = id     self.email = email     self.name = name      //now simply try to decode the optionals     self.profile = try? values.decode(String.self, forKey: .profile)     self.motive = try? values.decode(String.self, forKey: .motive)     self.address = try? values.decode(String.self, forKey: .address)     self.profilePhotoUrl = try? values.decode(String.self, forKey: .profilePhotoUrl) } 

SOMETHING TO NOTE: You should write your own encoder as well to stay consistent.

All of that can go, simply to a nice calling statement like this:

    if let user = try? JSONDecoder().decode(User.self, from: jsonData) {         //do stuff with user     } 

This is probably the safest, swift-ist, and most object oriented way to handle this type of issue.

Answers 9

I recommend to keep all non-scalar properties as optional, scalar as non-optional(with some exceptions) by assigning a default value and collections with empty array. e.g,

class User {     var id: Int = 0     var name: String?     var friends: [User] = [] } 

This assures you a crash free app no matter what happens to server. I would prefer abnormal behavior over a crash.

Answers 10

I would prefer optional properties because you can not promise JSON values to be there all the time and any change on response property name would crash your app.

If you do not use optional values, you have to control parameters while parsing and add a default value if you want a crash free app. And you wouldn't know if it was nil or empty string from server.

Optional values is your best friends.

object mapper for mutable and non-mutable properties.

realm-swift for default non-optional values.

Read More

JSON to CSV flattening nested JSON

I am trying to Flatten JSON to parse as a CSV. But the flattening is not properly flattening. When I get the json to flatten customer.addresses is filling with addresstype:r then skipping all fields city,countrycode,countycode etc. and then starting at customer.companyName. The nested JSON is not breaking up properly to show properly in excel I think my JavaScript code must be off just a little bit. Any help with this would be greatly appreciated.

JSON (this is a portion of the nested json it will not always be in the same depth is there a way to code for any type of nested json that will read at all levels)

[   {     "countyCode": 12,     "customer": {       "addresses": [         {           "addressType": "R",           "city": "BRADENTON",           "countryCode": "US",           "countyCode": 12,           "foreignPostalCode": null,           "state": "FL",           "streetAddress": "819 15th Ave Dr E",           "zipCode": 34211,           "zipPlus": null         },         {           "addressType": "M",           "city": "BRADENTON",           "countryCode": "US",           "countyCode": 12,           "foreignPostalCode": null,           "state": "FL",           "streetAddress": "PO BOX 124",           "zipCode": 34201,           "zipPlus": 0124         }       ],       "companyName": null,       "customerNumber": 932874,       "customerStopFlag": false,       "customerType": "I",       "dateOfBirth": "1936-08-05T00:00:00",       "dlExpirationDate": "2022-08-05T00:00:00",       "dlRenewalEligibilityFlag": true,       "driverLicenseNumber": "B360722339284",       "emailAddress": null,       "feidNumber": null,       "firstName": "David",       "lastName": "Brierton",       "middleName": "Hugh",       "militaryExemptionFlag": null,       "nameSuffix": null,       "sex": "M" 

JS

function flatObjectToString(obj) {     var s = "";     Object.keys(obj).map(key => {       if (obj[key] === null) {         s += key + ":";       } else if (obj[key].toLocaleDateString) {         s += key + ": " + obj[key].toLocaleDateString() + "\n";       } else if (obj[key] instanceof Array) {         s += key + ":\n" + listToFlatString(obj[key]);       } else if (typeof obj[key] == "object") {         s += key + ":\n" + flatObjectToString(obj[key]);       } else {         s += key + ":" + obj[key];       }       s += "\n";     });     return s;   }    function listToFlatString(list) {     var s = "";     list.map(item => {       Object.keys(item).map(key => {         s += "";         if (item[key] instanceof Array) {           s += key + "\n" + listToFlatString(item[key]);         } else if (typeof item[key] == "object" && item[key] !== null) {           s += key + ": " + flatObjectToString(item[key]);         } else {           s += key + ": " + (item[key] === null ? "" : item[key].toLocaleDateString ? item[key].toLocaleDateString : item[key].toString());         }         s += "\n";       });     });     return s;   }    function flatten(object, addToList, prefix) {     Object.keys(object).map(key => {       if (object[key] === null) {         addToList[prefix + key] = "";       } else       if (object[key] instanceof Array) {         addToList[prefix + key] = listToFlatString(object[key]);       } else if (typeof object[key] == 'object' && !object[key].toLocaleDateString) {         flatten(object[key], addToList, prefix + key + '.');       } else {         addToList[prefix + key] = object[key];       }     });     return addToList;   } 

Then I run it through the Javascript Utilities with this:

// Run the JSON string through the flattening utilities above           var flatJSON = JSON.parse(evt.target.result).map(record => flatten(record, {}, ''));            var csv = Papa.unparse(flatJSON); 

4 Answers

Answers 1

You can use something like below

data = require("./data.json")   flattenObject = (obj) => {     let flattenKeys = {};     for (let i in obj) {         if (!obj.hasOwnProperty(i)) continue;         if ((typeof obj[i]) == 'object') {             // flattenKeys[i] = obj[i];             let flatObject = flattenObject(obj[i]);             for (let j in flatObject) {                 if (!flatObject.hasOwnProperty(j)) continue;                 flattenKeys[i + '.' + j] = flatObject[j];             }         } else {             flattenKeys[i] = obj[i];         }     }     return flattenKeys; }  console.log(flattenObject(data)) 

The output on first element of your array object is as below

{ countyCode: 12,   'customer.addresses.0.addressType': 'R',   'customer.addresses.0.city': 'BRADENTON',   'customer.addresses.0.countryCode': 'US',   'customer.addresses.0.countyCode': 12,   'customer.addresses.0.state': 'FL',   'customer.addresses.0.streetAddress': '819 15th Ave Dr E',   'customer.addresses.0.zipCode': 34211,   'customer.addresses.1.addressType': 'M',   'customer.addresses.1.city': 'BRADENTON',   'customer.addresses.1.countryCode': 'US',   'customer.addresses.1.countyCode': 12,   'customer.addresses.1.state': 'FL',   'customer.addresses.1.streetAddress': 'PO BOX 124',   'customer.addresses.1.zipCode': 34201,   'customer.addresses.1.zipPlus': '124',   'customer.customerNumber': 932874,   'customer.customerStopFlag': false,   'customer.customerType': 'I',   'customer.dateOfBirth': '1936-08-05T00:00:00',   'customer.dlExpirationDate': '2022-08-05T00:00:00',   'customer.dlRenewalEligibilityFlag': true,   'customer.driverLicenseNumber': 'B360722339284',   'customer.firstName': 'David',   'customer.lastName': 'Brierton',   'customer.middleName': 'Hugh',   'customer.sex': 'M' } 

Answers 2

Another solution:

function flatObjectToString(obj) {     var path = [],         nodes = {},         parseObj = function (obj) {             if (typeof obj == 'object') {                 if (obj instanceof Array) { //array                     for (var i = 0, l = obj.length; i < l; i++) {                         path.push(i);                         parseObj(obj[i]);                         path.pop();                     }                 }                 else {  //object                     for (var node in obj) {                         path.push(node);                         parseObj(obj[node]);                         path.pop();                     }                 }             }             else {  //value                 nodes[path.join('_')] = obj;             }         };      parseObj(obj);     return nodes; }  console.log(JSON.stringify(flatObjectToString(data))); 

• I used a global path array to store the recursive path and prevent a lot a string operations
• I don't know about the size of the actual JSON files so I tried to get a solution which doesn't waste any garbage
• There is no need to check for "hasOwnProperty" in this (your) case

Answers 3

If you are looking for the fix to your actual code then you need to use something like below

function flatten(object, addToList, prefix) {     Object.keys(object).map(key => {         if (object[key] === null) {             addToList[prefix + key] = "";         } else         if (object[key] instanceof Array) {             // addToList[prefix + key] = listToFlatString(object[key]);             for (i in object[key]) {                 flatten(object[key][i], addToList, prefix + key + "." + i)             }         } else if (typeof object[key] == 'object' && !object[key].toLocaleDateString) {             flatten(object[key], addToList, prefix + key + '.');         } else {             addToList[prefix + key] = object[key];         }     });     return addToList; } 

I replace

addToList[prefix + key] = listToFlatString(object[key]); 

with

for (i in object[key]) {    flatten(object[key][i], addToList, prefix + key + "." + i) } 

That fixed the issue and the output is below as per the sample json you provided

[ '0.countyCode': 12,   '0.excludedFlag': '',   '0.fees.annualTotalFees': 35.6,   '0.fees.annualTotalFeesMail': 36.35,   '0.fees.annualTotalFeesOnline': 38.35,   '0.fees.biennialTotalFees': 71.2,   '0.fees.biennialTotalFeesMail': 71.95,   '0.fees.biennialTotalFeesOnline': 73.95,   '0.fees.branchFeeFlag': false,   '0.fees.delinquentFeeAmount': 5,   '0.fees.mhBackTax': 0,   '0.fees.mhBackTaxMonths': 0,   '0.fileId': 522,   '0.id': '0002be04-546-4a92-a3d7-c31546544f',   '0.messages.0messageCode': 'RN2',   '0.messages.0messageText': 'Plate must be replaced if a 2 year renewal is desired.',   '0.messages.0messageType': 'I',   '0.messages.1messageCode': 'RN40',   '0.messages.1messageText': 'You must complete the affidavit on the reverse side or provide a copy of your Florida insurance identification card.',   '0.messages.1messageType': 'I',   '0.messages.2messageCode': 'RN41',   '0.messages.2messageText': 'Insurance is on file.',   '0.messages.2messageType': 'II',   '0.registrationDetail.annualPlateReplacementFlag': false,   '0.registrationDetail.arfCredit': 25.6,   '0.registrationDetail.biennialPlateReplacementFlag': true,   '0.registrationDetail.customerStopFlag': '',   '0.registrationDetail.delinquentDate': '2018-02-11T00:00:00',   '0.registrationDetail.expirationDate': '2018-01-09T00:00:00',   '0.registrationDetail.foreignAddressFlag': false,   '0.registrationDetail.honorayConsulPlateFlag': '',   '0.registrationDetail.hovDecalNumber': '',   '0.registrationDetail.hovDecalYear': '',   '0.registrationDetail.hovExpirationDate': '',   '0.registrationDetail.inventorySubtype': 'RP',   '0.registrationDetail.lastActivityCounty': 12,   '0.registrationDetail.legislativePlateDueForReplacementFlag': false,   '0.registrationDetail.licensePlateCode': 'RGR',   '0.registrationDetail.licensePlateNumber': 'L45656',   '0.registrationDetail.mailToAddressFlag': false,   '0.registrationDetail.mailToCustomerNumber': '',   '0.registrationDetail.mhLocationCode': '',   '0.registrationDetail.militaryOwnerFlag': false,   '0.registrationDetail.numberOfRegistrants': 1,   '0.registrationDetail.onlineRenewalEligibilityFlag': true,   '0.registrationDetail.pinNumber': 222354654,   '0.registrationDetail.plateExpirationDate': '2018-03-18T00:00:00',   '0.registrationDetail.plateIssueDate': '2008-09-18T00:00:00',   '0.registrationDetail.possibleNgExemptionFlag': false,   '0.registrationDetail.registrationNumber': 2234545345,   '0.registrationDetail.registrationOnlyFlag': '',   '0.registrationDetail.registrationType': 'R',   '0.registrationDetail.registrationUse': 'PR',   '0.registrationDetail.renewalCountyCode': 12,   '0.registrationDetail.rentalParkFlag': '',   '0.registrationDetail.seminoleMiccosukeeIndianFlag': false,   '0.registrationDetail.taxCollectorRenewalEligibilityFlag': true,   '0.registrationDetail.vehicleClassCode': 1,   '0.registrationOwners.0customer.addresses.0addressType': 'R',   '0.registrationOwners.0customer.addresses.0city': 'PALMETTO',   '0.registrationOwners.0customer.addresses.0countryCode': 'US',   '0.registrationOwners.0customer.addresses.0countyCode': 12,   '0.registrationOwners.0customer.addresses.0foreignPostalCode': '',   '0.registrationOwners.0customer.addresses.0state': 'FL',   '0.registrationOwners.0customer.addresses.0streetAddress': '34545 7TH AVE W',   '0.registrationOwners.0customer.addresses.0zipCode': 34221,   '0.registrationOwners.0customer.addresses.0zipPlus': '',   '0.registrationOwners.0customer.addresses.1addressType': 'M',   '0.registrationOwners.0customer.addresses.1city': 'PALMETTO',   '0.registrationOwners.0customer.addresses.1countryCode': 'US',   '0.registrationOwners.0customer.addresses.1countyCode': 12,   '0.registrationOwners.0customer.addresses.1foreignPostalCode': '',   '0.registrationOwners.0customer.addresses.1state': 'FL',   '0.registrationOwners.0customer.addresses.1streetAddress': '34545 7TH AVE W',   '0.registrationOwners.0customer.addresses.1zipCode': 34221,   '0.registrationOwners.0customer.addresses.1zipPlus': 3433,   '0.registrationOwners.0customer.companyName': '',   '0.registrationOwners.0customer.customerNumber': 3346645,   '0.registrationOwners.0customer.customerStopFlag': false,   '0.registrationOwners.0customer.customerType': 'I',   '0.registrationOwners.0customer.dateOfBirth': '1971-01-09T00:00:00',   '0.registrationOwners.0customer.dlExpirationDate': '2025-01-09T00:00:00',   '0.registrationOwners.0customer.dlRenewalEligibilityFlag': true,   '0.registrationOwners.0customer.driverLicenseNumber': 'F34545345345',   '0.registrationOwners.0customer.emailAddress': '',   '0.registrationOwners.0customer.feidNumber': '',   '0.registrationOwners.0customer.firstName': 'DAVID',   '0.registrationOwners.0customer.lastName': 'HUGH',   '0.registrationOwners.0customer.middleName': 'BRIERTON',   '0.registrationOwners.0customer.militaryExemptionFlag': false,   '0.registrationOwners.0customer.nameSuffix': '',   '0.registrationOwners.0customer.sex': 'F',   '0.registrationOwners.0registrationOwnershipNumber': 1,   '0.shippingAddress.address.addressType': 'S',   '0.shippingAddress.address.city': 'PALMETTO',   '0.shippingAddress.address.countryCode': 'US',   '0.shippingAddress.address.countyCode': 12,   '0.shippingAddress.address.foreignPostalCode': '',   '0.shippingAddress.address.state': 'FL',   '0.shippingAddress.address.streetAddress': '34545 7TH AVE W',   '0.shippingAddress.address.zipCode': 34221,   '0.shippingAddress.address.zipPlus': 8344,   '0.shippingAddress.shippingCompanyName': '',   '0.shippingAddress.shippingName1': 'DAVID HUGH BRIERTON',   '0.shippingAddress.shippingName2': '',   '0.stops': '',   '0.vehicle.address': '',   '0.vehicle.bodyCode': '4D',   '0.vehicle.brakeHorsePower': '',   '0.vehicle.cubicCentimeters': '',   '0.vehicle.grossWeight': '',   '0.vehicle.insuranceAffidavitCode': '',   '0.vehicle.leaseOwnerFlag': false,   '0.vehicle.lengthFeet': '',   '0.vehicle.lengthInches': '',   '0.vehicle.majorColorCode': 'GLD',   '0.vehicle.makeCode': 'CHEV',   '0.vehicle.minorColorCode': '',   '0.vehicle.netWeight': 3200,   '0.vehicle.numberOfAxles': '',   '0.vehicle.ownerUnitNumber': '',   '0.vehicle.titleNumber': 345454534,   '0.vehicle.vehicleIdentificationNumber': '1G1ZD345U5B345435',   '0.vehicle.vehicleNumber': 23454656,   '0.vehicle.vehicleType': 'AU',   '0.vehicle.vesselCode': '',   '0.vehicle.vesselManufacturerDesc': '',   '0.vehicle.vesselResidentStatus': 'Y',   '0.vehicle.vesselWaterType': '',   '0.vehicle.widthFeet': '',   '0.vehicle.widthInches': '',   '0.vehicle.yearMake': 2011,   '1.countyCode': 12,   '1.excludedFlag': '',   '1.fees.annualTotalFees': 27.6,   '1.fees.annualTotalFeesMail': 28.35,   '1.fees.annualTotalFeesOnline': 30.35,   '1.fees.biennialTotalFees': 55.2,   '1.fees.biennialTotalFeesMail': 55.95,   '1.fees.biennialTotalFeesOnline': 57.95,   '1.fees.branchFeeFlag': false,   '1.fees.delinquentFeeAmount': 5,   '1.fees.mhBackTax': 0,   '1.fees.mhBackTaxMonths': 0,   '1.fileId': 522,   '1.id': '0008c654-8960-45b8-b416-cff3456767',   '1.messages.0messageCode': 'RN40',   '1.messages.0messageText': 'You must complete the affidavit on the reverse side or provide a copy of your Florida insurance identification card.',   '1.messages.0messageType': 'I',   '1.registrationDetail.annualPlateReplacementFlag': false,   '1.registrationDetail.arfCredit': 2.8,   '1.registrationDetail.biennialPlateReplacementFlag': false,   '1.registrationDetail.customerStopFlag': '',   '1.registrationDetail.delinquentDate': '2018-02-11T00:00:00',   '1.registrationDetail.expirationDate': '2018-01-01T00:00:00',   '1.registrationDetail.foreignAddressFlag': false,   '1.registrationDetail.honorayConsulPlateFlag': '',   '1.registrationDetail.hovDecalNumber': '',   '1.registrationDetail.hovDecalYear': '',   '1.registrationDetail.hovExpirationDate': '',   '1.registrationDetail.inventorySubtype': 'SS',   '1.registrationDetail.lastActivityCounty': 16,   '1.registrationDetail.legislativePlateDueForReplacementFlag': false,   '1.registrationDetail.licensePlateCode': 'RGS',   '1.registrationDetail.licensePlateNumber': 'HU34598',   '1.registrationDetail.mailToAddressFlag': false,   '1.registrationDetail.mailToCustomerNumber': '',   '1.registrationDetail.mhLocationCode': '',   '1.registrationDetail.militaryOwnerFlag': false,   '1.registrationDetail.numberOfRegistrants': 1,   '1.registrationDetail.onlineRenewalEligibilityFlag': true,   '1.registrationDetail.pinNumber': 4936856,   '1.registrationDetail.plateExpirationDate': '2026-09-24T00:00:00',   '1.registrationDetail.plateIssueDate': '2017-03-24T00:00:00',   '1.registrationDetail.possibleNgExemptionFlag': false,   '1.registrationDetail.registrationNumber': 4095685,   '1.registrationDetail.registrationOnlyFlag': '',   '1.registrationDetail.registrationType': 'R',   '1.registrationDetail.registrationUse': 'PR',   '1.registrationDetail.renewalCountyCode': 12,   '1.registrationDetail.rentalParkFlag': '',   '1.registrationDetail.seminoleMiccosukeeIndianFlag': false,   '1.registrationDetail.taxCollectorRenewalEligibilityFlag': true,   '1.registrationDetail.vehicleClassCode': 1,   '1.registrationOwners.0customer.addresses.0addressType': 'R',   '1.registrationOwners.0customer.addresses.0city': 'SARASOTA',   '1.registrationOwners.0customer.addresses.0countryCode': 'US',   '1.registrationOwners.0customer.addresses.0countyCode': 12,   '1.registrationOwners.0customer.addresses.0foreignPostalCode': '',   '1.registrationOwners.0customer.addresses.0state': 'FL',   '1.registrationOwners.0customer.addresses.0streetAddress': '5858 FRUITVILLE RD',   '1.registrationOwners.0customer.addresses.0zipCode': 34240,   '1.registrationOwners.0customer.addresses.0zipPlus': 5858,   '1.registrationOwners.0customer.addresses.1addressType': 'M',   '1.registrationOwners.0customer.addresses.1city': 'SARASOTA',   '1.registrationOwners.0customer.addresses.1countryCode': 'US',   '1.registrationOwners.0customer.addresses.1countyCode': 16,   '1.registrationOwners.0customer.addresses.1foreignPostalCode': '',   '1.registrationOwners.0customer.addresses.1state': 'FL',   '1.registrationOwners.0customer.addresses.1streetAddress': '5858 FRUITVILLE RD',   '1.registrationOwners.0customer.addresses.1zipCode': 34240,   '1.registrationOwners.0customer.addresses.1zipPlus': 5858,   '1.registrationOwners.0customer.companyName': '',   '1.registrationOwners.0customer.customerNumber': 2928357,   '1.registrationOwners.0customer.customerStopFlag': false,   '1.registrationOwners.0customer.customerType': 'I',   '1.registrationOwners.0customer.dateOfBirth': '1989-01-01T00:00:00',   '1.registrationOwners.0customer.dlExpirationDate': '2022-01-01T00:00:00',   '1.registrationOwners.0customer.dlRenewalEligibilityFlag': true,   '1.registrationOwners.0customer.driverLicenseNumber': 'B94832734',   '1.registrationOwners.0customer.emailAddress': '',   '1.registrationOwners.0customer.feidNumber': '',   '1.registrationOwners.0customer.firstName': 'DAVID1',   '1.registrationOwners.0customer.lastName': 'HUGH1',   '1.registrationOwners.0customer.middleName': 'BRIERTON1',   '1.registrationOwners.0customer.militaryExemptionFlag': false,   '1.registrationOwners.0customer.nameSuffix': '',   '1.registrationOwners.0customer.sex': 'M',   '1.registrationOwners.0registrationOwnershipNumber': 1,   '1.shippingAddress.address.addressType': 'S',   '1.shippingAddress.address.city': 'SARASOTA',   '1.shippingAddress.address.countryCode': 'US',   '1.shippingAddress.address.countyCode': 16,   '1.shippingAddress.address.foreignPostalCode': '',   '1.shippingAddress.address.state': 'FL',   '1.shippingAddress.address.streetAddress': '5858 FRUITVILLE RD',   '1.shippingAddress.address.zipCode': 34240,   '1.shippingAddress.address.zipPlus': 5858,   '1.shippingAddress.shippingCompanyName': '',   '1.shippingAddress.shippingName1': 'DAVID1 HUGH1 BRIERTON1',   '1.shippingAddress.shippingName2': '',   '1.stops': '',   '1.vehicle.address': '',   '1.vehicle.bodyCode': '4D',   '1.vehicle.brakeHorsePower': '',   '1.vehicle.cubicCentimeters': '',   '1.vehicle.grossWeight': '',   '1.vehicle.insuranceAffidavitCode': '',   '1.vehicle.leaseOwnerFlag': false,   '1.vehicle.lengthFeet': '',   '1.vehicle.lengthInches': '',   '1.vehicle.majorColorCode': 'BLU',   '1.vehicle.makeCode': 'STRN',   '1.vehicle.minorColorCode': '',   '1.vehicle.netWeight': 2290,   '1.vehicle.numberOfAxles': '',   '1.vehicle.ownerUnitNumber': '',   '1.vehicle.titleNumber': 239874,   '1.vehicle.vehicleIdentificationNumber': '1G832492871Z23094',   '1.vehicle.vehicleNumber': 239084,   '1.vehicle.vehicleType': 'AU',   '1.vehicle.vesselCode': '',   '1.vehicle.vesselManufacturerDesc': '',   '1.vehicle.vesselResidentStatus': 'Y',   '1.vehicle.vesselWaterType': '',   '1.vehicle.widthFeet': '',   '1.vehicle.widthInches': '',   '1.vehicle.yearMake': 2001,   '2.countyCode': 12,   '2.excludedFlag': '',   '2.fees.annualTotalFees': 45.6,   '2.fees.annualTotalFeesMail': 46.35,   '2.fees.annualTotalFeesOnline': 48.35,   '2.fees.biennialTotalFees': 91.2,   '2.fees.biennialTotalFeesMail': 91.95,   '2.fees.biennialTotalFeesOnline': 93.95,   '2.fees.branchFeeFlag': false,   '2.fees.delinquentFeeAmount': 10,   '2.fees.mhBackTax': 0,   '2.fees.mhBackTaxMonths': 0,   '2.fileId': 522,   '2.id': '000e3450d-3454-499a-ae70-de5676577',   '2.messages.0messageCode': 'RN40',   '2.messages.0messageText': 'You must complete the affidavit on the reverse side or provide a copy of your Florida insurance identification card.',   '2.messages.0messageType': 'I',   '2.registrationDetail.annualPlateReplacementFlag': false,   '2.registrationDetail.arfCredit': 8.4,   '2.registrationDetail.biennialPlateReplacementFlag': false,   '2.registrationDetail.customerStopFlag': '',   '2.registrationDetail.delinquentDate': '2018-02-11T00:00:00',   '2.registrationDetail.expirationDate': '2018-01-11T00:00:00',   '2.registrationDetail.foreignAddressFlag': false,   '2.registrationDetail.honorayConsulPlateFlag': '',   '2.registrationDetail.hovDecalNumber': '',   '2.registrationDetail.hovDecalYear': '',   '2.registrationDetail.hovExpirationDate': '',   '2.registrationDetail.inventorySubtype': 'RP',   '2.registrationDetail.lastActivityCounty': 12,   '2.registrationDetail.legislativePlateDueForReplacementFlag': false,   '2.registrationDetail.licensePlateCode': 'RGR',   '2.registrationDetail.licensePlateNumber': '808IUT',   '2.registrationDetail.mailToAddressFlag': false,   '2.registrationDetail.mailToCustomerNumber': '',   '2.registrationDetail.mhLocationCode': '',   '2.registrationDetail.militaryOwnerFlag': false,   '2.registrationDetail.numberOfRegistrants': 1,   '2.registrationDetail.onlineRenewalEligibilityFlag': true,   '2.registrationDetail.pinNumber': 934597,   '2.registrationDetail.plateExpirationDate': '2023-06-06T00:00:00',   '2.registrationDetail.plateIssueDate': '2013-12-06T00:00:00',   '2.registrationDetail.possibleNgExemptionFlag': false,   '2.registrationDetail.registrationNumber': 39287432,   '2.registrationDetail.registrationOnlyFlag': '',   '2.registrationDetail.registrationType': 'R',   '2.registrationDetail.registrationUse': 'PR',   '2.registrationDetail.renewalCountyCode': 12,   '2.registrationDetail.rentalParkFlag': '',   '2.registrationDetail.seminoleMiccosukeeIndianFlag': false,   '2.registrationDetail.taxCollectorRenewalEligibilityFlag': true,   '2.registrationDetail.vehicleClassCode': 1,   '2.registrationOwners.0customer.addresses.0addressType': 'R',   '2.registrationOwners.0customer.addresses.0city': 'SARASOTA',   '2.registrationOwners.0customer.addresses.0countryCode': 'US',   '2.registrationOwners.0customer.addresses.0countyCode': 12,   '2.registrationOwners.0customer.addresses.0foreignPostalCode': '',   '2.registrationOwners.0customer.addresses.0state': 'FL',   '2.registrationOwners.0customer.addresses.0streetAddress': '39875 44TH DR E',   '2.registrationOwners.0customer.addresses.0zipCode': 34243,   '2.registrationOwners.0customer.addresses.0zipPlus': 5566,   '2.registrationOwners.0customer.addresses.1addressType': 'M',   '2.registrationOwners.0customer.addresses.1city': 'PALMETTO',   '2.registrationOwners.0customer.addresses.1countryCode': 'US',   '2.registrationOwners.0customer.addresses.1countyCode': 12,   '2.registrationOwners.0customer.addresses.1foreignPostalCode': '',   '2.registrationOwners.0customer.addresses.1state': 'FL',   '2.registrationOwners.0customer.addresses.1streetAddress': '39875 44TH DR E',   '2.registrationOwners.0customer.addresses.1zipCode': 34221,   '2.registrationOwners.0customer.addresses.1zipPlus': '',   '2.registrationOwners.0customer.companyName': '',   '2.registrationOwners.0customer.customerNumber': 2398574,   '2.registrationOwners.0customer.customerStopFlag': false,   '2.registrationOwners.0customer.customerType': 'I',   '2.registrationOwners.0customer.dateOfBirth': '1958-01-11T00:00:00',   '2.registrationOwners.0customer.dlExpirationDate': '2020-01-11T00:00:00',   '2.registrationOwners.0customer.dlRenewalEligibilityFlag': true,   '2.registrationOwners.0customer.driverLicenseNumber': 'B23987433',   '2.registrationOwners.0customer.emailAddress': '',   '2.registrationOwners.0customer.feidNumber': '',   '2.registrationOwners.0customer.firstName': 'DAVID2',   '2.registrationOwners.0customer.lastName': 'HUGH2',   '2.registrationOwners.0customer.middleName': 'BRIERTON2',   '2.registrationOwners.0customer.militaryExemptionFlag': false,   '2.registrationOwners.0customer.nameSuffix': '',   '2.registrationOwners.0customer.sex': 'M',   '2.registrationOwners.0registrationOwnershipNumber': 1,   '2.shippingAddress.address.addressType': 'S',   '2.shippingAddress.address.city': 'PALMETTO',   '2.shippingAddress.address.countryCode': 'US',   '2.shippingAddress.address.countyCode': 12,   '2.shippingAddress.address.foreignPostalCode': '',   '2.shippingAddress.address.state': 'FL',   '2.shippingAddress.address.streetAddress': '293847 33TH ST W',   '2.shippingAddress.address.zipCode': 34221,   '2.shippingAddress.address.zipPlus': '',   '2.shippingAddress.shippingCompanyName': '',   '2.shippingAddress.shippingName1': 'DAVID2 HUGH2 BRIERTON2',   '2.shippingAddress.shippingName2': '',   '2.stops': '',   '2.vehicle.address': '',   '2.vehicle.bodyCode': '2D',   '2.vehicle.brakeHorsePower': '',   '2.vehicle.cubicCentimeters': '',   '2.vehicle.grossWeight': '',   '2.vehicle.insuranceAffidavitCode': '',   '2.vehicle.leaseOwnerFlag': false,   '2.vehicle.lengthFeet': '',   '2.vehicle.lengthInches': '',   '2.vehicle.majorColorCode': 'BLK',   '2.vehicle.makeCode': 'PONT',   '2.vehicle.minorColorCode': '',   '2.vehicle.netWeight': 3802,   '2.vehicle.numberOfAxles': '',   '2.vehicle.ownerUnitNumber': '',   '2.vehicle.titleNumber': 239857424,   '2.vehicle.vehicleIdentificationNumber': '6G23242312UX63297437',   '2.vehicle.vehicleNumber': '35T7843',   '2.vehicle.vehicleType': 'AU',   '2.vehicle.vesselCode': '',   '2.vehicle.vesselManufacturerDesc': '',   '2.vehicle.vesselResidentStatus': 'Y',   '2.vehicle.vesselWaterType': '',   '2.vehicle.widthFeet': '',   '2.vehicle.widthInches': '',   '2.vehicle.yearMake': 2006 ] 

Answers 4

I would reccomend using flat and not re-inventing the wheel. If you really don't want to do that you could just read Hugh's code here. He supports both flatten and unflatten. Here is the flatten function for reference:

var isBuffer = require('is-buffer')    module.exports = flatten  flatten.flatten = flatten  flatten.unflatten = unflatten    function flatten (target, opts) {    opts = opts || {}      var delimiter = opts.delimiter || '.'    var maxDepth = opts.maxDepth    var output = {}      function step (object, prev, currentDepth) {      currentDepth = currentDepth || 1      Object.keys(object).forEach(function (key) {        var value = object[key]        var isarray = opts.safe && Array.isArray(value)        var type = Object.prototype.toString.call(value)        var isbuffer = isBuffer(value)        var isobject = (          type === '[object Object]' ||          type === '[object Array]'        )          var newKey = prev          ? prev + delimiter + key          : key          if (!isarray && !isbuffer && isobject && Object.keys(value).length &&          (!opts.maxDepth || currentDepth < maxDepth)) {          return step(value, newKey, currentDepth + 1)        }          output[newKey] = value      })    }      step(target)      return output  }

Read More