Hive Sql dynamically get null column counts from a table

I am using datastax + spark integration and spark SQL thrift server, which gives me a Hive SQL interface to query the tables in Cassandra.

The tables in my database get dynamically created, what I want to do is get a count of null values in each column for the table based on just the table name.

I can get the column names using describe database.table but in hive SQL, how do I use its output in another select query which counts null for all the columns.

Update 1: Traceback with Dudu's solution

Error running query: TExecuteStatementResp(status=TStatus(errorCode=0, errorMessage="org.apache.spark.sql.AnalysisException: Invalid usage of '*' in explode/json_tuple/UDTF;", sqlState=None, infoMessages=["org.apache.hive.service.cli.HiveSQLException:org.apache.spark.sql.AnalysisException: Invalid usage of '' in explode/json_tuple/UDTF;:16:15", 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:258', 'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:runInternal:SparkExecuteStatementOperation.scala:152', 'org.apache.hive.service.cli.operation.Operation:run:Operation.java:257', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:388', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:369', 'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:262', 'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:437', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1313', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1298', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:617', 'java.lang.Thread:run:Thread.java:745'], statusCode=3), operationHandle=None)

3 Answers

Answers 1

In the following solution there is no need to deal with each column separately. The result is a column index and the number of null values in that column.
You can later on join it by the column index to an information retrieved from the metastore.
One limitations is that strings containning the exact text null will be counted as nulls.

Demo

The CTE (mytable as defined by with mytable as) can obviously be replaced by as actual table

with        mytable as              (                 select  stack                         (                             5                             ,1   ,1.2     ,date '2017-06-21'     ,null                            ,2   ,2.3     ,null                  ,null                            ,3   ,null    ,null                  ,'hello'                            ,4   ,4.5     ,null                  ,'world'                            ,5   ,null    ,date '2017-07-22'     ,null                         ) as (id,amt,dt,txt)             )  select      pe.pos                                          as col_index            ,count(case when pe.val='null' then 1 end)       as nulls_count  from        mytable t lateral view posexplode (split(printf(concat('%s',repeat('\u0001%s',field(unhex(1),t.*,unhex(1))-2)),t.*),'\\x01')) pe  group by    pe.pos        ; 

---

+-----------+-------------+ | col_index | nulls_count | +-----------+-------------+ |         0 |           0 | |         1 |           2 | |         2 |           3 | |         3 |           3 | +-----------+-------------+ 

Answers 2

Instead of describe database.table, you can use

Select column_name from system_schema.columns where keyspace_name='YOUR KEYSPACE' and table_name='YOUR TABLE'

There is also a column called kind in the above table with values like partition_key,clustering,regular.

The columns with values as partition_key and clustering will not have null values.

For other columns you can use

select sum(CASE WHEN col1 is NULL THEN 1 ELSE 0 END) as col1_cnt,sum(CASE WHEN col2 is NULL THEN 1 ELSE 0 END) as col2_cnt from table1 where col1 is null;

You can also try below query (Not tried myself)

SELECT COUNT(*)-COUNT(col1) As A, COUNT(*)-COUNT(col2) As B, COUNT(*)-COUNT(col3) As C FROM YourTable;  

May be for above query you can create variable for count instead of count(*) everytime.

Note: system_schema.columns is cassandra table and cassandra user should have read permission to this table

Answers 3

You will have to count null values from each column separately. For example -

select count(*) from mytable where col1 is null; select count(*) from mytable where col2 is null; 

Read More

How to iterate over array in ionize-cms with codeigniter?

I'm using Ionize cms for the back end of my site and i want to create my own Tags - for passing data from my own tables. I've followed This tutorial to create custom tags, and by that - passing data to views, but i keep getting error :

Tag missing: demo, scope: .

Here is my view :

<ul>     <ion:demo:details>          <li><ion:detail field="user_name" /></li>     </ion:demo:details> </ul> 

And here are the changes I've added to TagManager.php

 public static $tag_definitions = array     (     // <ion:demo:authors /> calls the method “tag_details”     "demo:details" =>      "tag_details",     "demo:details:detail" =>    "tag_detail",     ); 

I've also tried to create a simple codeigniter controller and pass the data with view() and to do something like :

<ul>    <?php foreach($details as $detail): ?>      <li>      <?php echo $detail['name'] ?>      </li>     <?php endforeach ;?> </ul> 

But i'm getting undefined error of $details...

1 Answers

Answers 1

Got it... changed the tags and now it's good.

<?php   class TagManager_Data extends TagManager  { /**  * Tags declaration  * To be available, each tag must be declared in this static array.  *  * @var array  *  */ public static $tag_definitions = array (     // <ion:demo:authors /> calls the method “tag_authors”     "authors" =>      "tag_authors",     "authors:author" =>    "tag_author",      );      /**      * Base module tag    * The index function of this class refers to the <ion:#module_name /> tag    * In other words, this function makes the <ion:#module_name /> tag    * available as main module parent tag for all other tags defined  * in this class.  *  * @usage  <ion:demo >  *      ...  *    </ion:demo>  *     */    public static function index(FTL_Binding $tag)    {     $str = $tag->expand();      return $str;     }     /**  * Loops through authors  *  * @param FTL_Binding $tag  * @return string  *  * @usage  <ion:demo:authors >  *        ...  *    </ion:demo:authors>  *  */    public static function tag_authors(FTL_Binding $tag) {     // Returned string     $str = '';      // Model load     //self::load_model('demo_author_model', 'author_model');      // Authors array     $authors = [["name"=>'josh',"foo"=>'josh'],     ["name"=>'joshjosh',"foo"=>'josh'],["name"=>'josh',"foo"=>'josh']];      foreach($authors as $author)     {         // Set the local tag var "author"         $tag->set('author', $author);          // Tag expand : Process of the children tags         $str .= $tag->expand();     }      return $str;    }    /**  * Author tag  *  * @param    FTL_Binding    Tag object  * @return    String      Tag attribute or ''  *  * @usage    <ion:demo:authors>  *        <ion:author field="name" />  *       </ion:demo:authors>  *  */ public static function tag_author(FTL_Binding $tag) {     // Returns the field value or NULL if the attribute is not set     $field = $tag->getAttribute('field');      if ( ! is_null($field))     {         $author = $tag->get('author');          if ( ! empty($author[$field]))         {             return self::output_value($tag, $author[$field]);         }          // Here we have the choice :         // - Ether return nothing if the field attribute isn't set or          doesn't exist         // - Ether silently return ''         return self::show_tag_error(             $tag,             'The attribute <b>"field"</b> is not set'         );         // return '';     }     }   } 

Read More

Wrong Leave deduction in Payslip odoo

I tried to generate payslip for an employee. An Employee has taken half day leave(0.5) but while calculating payslip its auto filled as 1.

From the code that is already in the module hr_payroll.py, It looks like the following

 def get_worked_day_lines(self, cr, uid, contract_ids, date_from, date_to, context=None):         """         @param contract_ids: list of contract id         @return: returns a list of dict containing the input that should be applied for the given contract between date_from and date_to         """         def was_on_leave(employee_id, datetime_day, context=None):             res = False             day = datetime_day.strftime("%Y-%m-%d")             holiday_ids = self.pool.get('hr.holidays').search(cr, uid, [('state','=','validate'),('employee_id','=',employee_id),('type','=','remove'),('date_from','<=',day),('date_to','>=',day)])             if holiday_ids:                 res = self.pool.get('hr.holidays').browse(cr, uid, holiday_ids, context=context)[0].holiday_status_id.name             return res          res = []         for contract in self.pool.get('hr.contract').browse(cr, uid, contract_ids, context=context):             if not contract.working_hours:                 #fill only if the contract as a working schedule linked                 continue             attendances = {                  'name': _("Normal Working Days paid at 100%"),                  'sequence': 1,                  'code': 'WORK100',                  'number_of_days': 0.0,                  'number_of_hours': 0.0,                  'contract_id': contract.id,             }             leaves = {}             day_from = datetime.strptime(date_from,"%Y-%m-%d")             day_to = datetime.strptime(date_to,"%Y-%m-%d")             nb_of_days = (day_to - day_from).days + 1             for day in range(0, nb_of_days):                 working_hours_on_day = self.pool.get('resource.calendar').working_hours_on_day(cr, uid, contract.working_hours, day_from + timedelta(days=day), context)                 if working_hours_on_day:                     #the employee had to work                     leave_type = was_on_leave(contract.employee_id.id, day_from + timedelta(days=day), context=context)                     if leave_type:                         #if he was on leave, fill the leaves dict                         if leave_type in leaves:                             leaves[leave_type]['number_of_days'] += 1.0                             leaves[leave_type]['number_of_hours'] += working_hours_on_day                         else:                             leaves[leave_type] = {                                 'name': leave_type,                                 'sequence': 5,                                 'code': leave_type,                                 'number_of_days': 1.0,                                 'number_of_hours': working_hours_on_day,                                 'contract_id': contract.id,                             }                     else:                         #add the input vals to tmp (increment if existing)                         attendances['number_of_days'] += 1.0                         attendances['number_of_hours'] += working_hours_on_day             leaves = [value for key,value in leaves.items()]             res += [attendances] + leaves         return res 

I am not sure whether this is where the issue is. Any one with any suggestion on this?

1 Answers

Answers 1

Let me preface: I haven't used odoo and I don't have the rest of your code to test against so I haven't verified this.

You definitely have a problem here:

if leave_type in leaves:     leaves[leave_type]['number_of_days'] += 1.0     leaves[leave_type]['number_of_hours'] += working_hours_on_day else:     leaves[leave_type] = {         'name': leave_type,         'sequence': 5,         'code': leave_type,         'number_of_days': 1.0,         'number_of_hours': working_hours_on_day,         'contract_id': contract.id,     } else:     #add the input vals to tmp (increment if existing)     attendances['number_of_days'] += 1.0     attendances['number_of_hours'] += working_hours_on_day 

see where you reference leaves and attendances, while those are floats they are only incremented in full days, not calculated based on what fraction of a day you pass. You need to change this to something like:

leaves[leave_type]['number_of_days'] += time_off / hours_per_workday # the 1.0 might make sense here depending on if you count time off as an attendance leaves[leave_type]['number_of_hours'] += time_off 

and

attendances['number_of_days'] += 1.0 - time_off / hours_per_workday attendances['number_of_hours'] += working_hours_on_day - time_off 

Obviously the dummy variables I inserted would need to be defined and calculated somewhere.

Additionally, as Anthony Rossi noted in the comments, you generally work in days, not hours. Examples:

day_from = datetime.strptime(date_from,"%Y-%m-%d") day_to = datetime.strptime(date_to,"%Y-%m-%d") 

notice how you only have YY/MM/DD and no hours.

Read More

Android TTS checking for supported locale with missing/not downloaded voice data

I'm using Android's TextToSpeech class. Everything is working normally. However, there are languages/locales that aren't installed by default but supported by the TTS engine and I can't capture the state of missing voice data.

With the internet on, when I try to setLanguage to a new locale which its voice data hasn't been downloaded, it'll simply download the voice data and perform the speak method normally/successfully.

However, with internet off, when I try to setLanguage to a new locale which its voice data hasn't been downloaded, it attempts to download the voice data. But with no internet, it just indicates "downloading" on the "TTS voice data" settings screen under "Language and input" for the selected locale, without any progress. And as expected the speak method doesn't work since the voice data isn't downloaded. When this happens, I would think TTS methods setLanguage/isLanguageAvailable will return LANG_MISSING_DATA for me to capture this state, however, it simply returns LANG_COUNTRY_AVAILABLE. The situation is shown in this image:

I want to be able to detect when the voice data of the locale being chosen isn't downloaded/missing and either give a toast message or direct user to download it. I have seen several posts suggesting the use of using isLanguageAvailable like this one. I also looked at the android documentation and it seems like isLanguageAvailable's return values should capture the state of missing voice data with LANG_MISSING_DATA.

I also tried sending an intent with ACTION_CHECK_TTS_DATA as the other way to check for missing data as suggested in the Android documentation I linked. However, the resultCode again didn't capture/indicate that the voice data is missing (CHECK_VOICE_DATA_FAIL) but returned CHECK_VOICE_DATA_PASS instead.

In this case, how should I capture the state of a language/locale being available/supported, with the voice data missing? I'm also curious why CHECK_VOICE_DATA_PASS and LANG_MISSING_DATA aren't the values returned. When the voice data is missing, shouldn't it return these values? Thanks! Below is the return value when I try to use setLanguage and isLanguageAvailable on locales that haven't had its voice data downloaded (0 and 1 are the returned value of the method shown in the logs, -1 is the one that corresponds to missing voice data):

1 Answers

Answers 1

You can find all available Locale of the device using following function. hope this code will help you.

 Locale loc = new Locale("en");  Locale[] availableLocales= loc.getAvailableLocales();  Boolean available=Boolean.FALSE;  for (int i=0;i<availableLocales.length;i++)  {   if(availableLocales[i].getDisplayLanguage().equals("your_locale_language"))    {         available=Boolean.TRUE;         // TODO:      }  } 

Read More

Wich is the most efficient way to iterate a directory?

Say I have a directory foo, with some number of subdirectories. Each of these subdirectories has between 0 and 5 files of variable length which I would like to process. My initial code looks like so:

    pool.query(`       SET SEARCH_PATH TO public,os_local;     `).then(() => fs.readdirSync(srcpath)         .filter(file => fs.lstatSync(path.join(srcpath, file)).isDirectory())         .map(dir => {           fs.access(`${srcpath + dir}/${dir}_Building.shp`, fs.constants.R_OK, (err) => {             if (!err) {               openShapeFile(`${srcpath + dir}/${dir}_Building.shp`).then((source) => source.read() .then(function dbWrite (result) {               if (result.done) {                 console.log(`done ${dir}`)               } else {     const query = `INSERT INTO os_local.buildings(geometry,                   id,                   featcode,                   version)                   VALUES(os_local.ST_GeomFromGeoJSON($1),                   $2,                   $3,                   $4) ON CONFLICT (id) DO UPDATE SET                     featcode=$3,                     geometry=os_local.ST_GeomFromGeoJSON($1),                     version=$4;`                 return pool.connect().then(client => {                   client.query(query, [geoJson.split('"[[').join('[[').split(']]"').join(']]'),                     result.value.properties.ID,                     result.value.properties.FEATCODE,                     version                   ]).then((result) => {                     return source.read().then(dbWrite)                   }).catch((err) => {                     console.log(err,                       query,                       geoJson.split('"[[').join('[[').split(']]"').join(']]'),                       result.value.properties.ID,                       result.value.properties.FEATCODE,                       version                     )                     return source.read().then(dbWrite)                   })                   client.release()                 })               }             })).catch(err => console.log('No Buildings', err))             }           })            fs.access(`${srcpath + dir}/${dir}__ImportantBuilding.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })            fs.access(`${srcpath + dir}/${dir}_Road.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })            fs.access(`${srcpath + dir}/${dir}_Glasshouse.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })            fs.access(`${srcpath + dir}/${dir}_RailwayStation.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })         }) 

This mostly works, but it ends up having to wait for the longest file to be fully processed in every subdirectory, resulting in practice in there always being only 1 connection to the database.

Is there a way I could rearchitect this to make better use of my computational resources, while limiting the number of active postgres connections and forcing code to wait until connections become available? (I set them to 20 in the pg poolConfig for node-postgres)

2 Answers

Answers 1

If you need to have your files processed in turn for a certain amount of time, then you can use Streams, timers(for scheduling) and process.nextTick(). There is great manual for understanding streams in nodejs.

Answers 2

Here is an example of getting directory contents using generators. You can start getting the first couple files right away and then use asynchronous code afterward to process files in parallel.

// Dependencies const fs = require('fs'); const path = require('path');  // The generator function (note the asterisk) function* getFilesInDirectory(fullPath, recursive = false) {     // Convert file names to full paths     let contents = fs.readdirSync(fullPath).map(file => {         return path.join(fullPath, file);     });      for(let i = 0; i < contents.length; i++) {         const childPath = contents[i];         let stats = fs.statSync(childPath);         if (stats.isFile()) {             yield childPath;         } else if (stats.isDirectory() && recursive) {             yield* getFilesInDirectory(childPath, true);         }     } } 

Usage:

function handleResults(results) {     ... // Returns a promise }  function processFile(file) {     ... // Returns a promise }  var files = getFilesInDirectory(__dirname, true); var result = files.next(); var promises = []; while(!result.done) {     console.log(result.value);     file = files.next();     // Process files in parallel     var promise = processFile(file).then(handleResults);     promises.push(promise); }  promise.all(promises).then() {     console.log(done); } 

Read More

elasticsearch: distributing indices over multiple disk volumes

I have one index which is quite large (about 100Gb), so I had to extend my disk space on my digital ocean survey by adding another volume (I run everything on only one node). I told elasticsearch that it now has to consider two disk locations by

/usr/share/elasticsearch/bin/elasticsearch -Epath.data=/var/lib/elasticsearch,/mnt/volume-sfo2-01/es_data 

elasticsearch does seem to have taken notice of this since it wrote some stuff to the new location

/mnt/volume-sfo2-01/es_data# cd nodes/ /mnt/volume-sfo2-01/es_data/nodes# ls 0 /mnt/volume-sfo2-01/es_data/nodes# cd 0/ /mnt/volume-sfo2-01/es_data/nodes/0# ls indices  node.lock  _state /mnt/volume-sfo2-01/es_data/nodes/0# cd indices /mnt/volume-sfo2-01/es_data/nodes/0/indices# ls DixLGLrJRXm1gSYcFzkzzw  nmZbce8wTayJC2s_eMC0-g  Qd-9ZnFIRoSM2z7AohKm-w  Sm_tyYTJTty0ImvDamFaQw /mnt/volume-sfo2-01/es_data/nodes/0/indices# cd DixLGLrJRXm1gSYcFzkzzw/ /mnt/volume-sfo2-01/es_data/nodes/0/indices/DixLGLrJRXm1gSYcFzkzzw# ls _state 

which is identical to the stuff I find in /var/lib/elasticsearch/data, except of the actual index information in the lowest level.

Reading the elasticsearch documentary I got the impression that elasticsearch is arranging the new index over the two disk locations, but will not split a shard between the two locations. So I initialized the index with 5 shards so that it can split the data between the volumes.

The survey does seem to have detected the two data paths since the log file shows

[2017-06-17T19:16:57,079][INFO ][o.e.e.NodeEnvironment    ] [WU6cQ-o] using [2] data paths, mounts [[/ (/dev/vda1), /mnt/volume-sfo2-01 (/dev/sda)]], net usable_space [29.6gb], net total_space [98.1gb], spins? [possibly], types [ext4] 

However, when I index the new indices, with constantly uses all the disk space on my original disk and eventually runs out of disk space with the error

raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) elasticsearch.exceptions.TransportError: TransportError(500, u'index_failed_engine_exception', u'Index failed for [pubmed_paper#25949809]') 

It never shifts one of the shards to the second volume? Do I miss anything? Can I manually guide the disk space usage?

Here are the elasticsearch version details:

# curl -XGET 'localhost:9200' {   "name" : "WU6cQ-o",   "cluster_name" : "elasticsearch",   "cluster_uuid" : "hKc147QfQqCefLliStLNtw",   "version" : {     "number" : "5.1.1",     "build_hash" : "5395e21",     "build_date" : "2016-12-06T12:36:15.409Z",     "build_snapshot" : false,     "lucene_version" : "6.3.0"   },   "tagline" : "You Know, for Search" } 

and here is the default path file structure, where ekasticsearch stores all the information (instead of sharing it with the second path)

/var/lib/elasticsearch/elasticsearch/nodes/0/indices/DixLGLrJRXm1gSYcFzkzzw# ls 0  1  2  3  4  _state 

one question is probably whether I can just take one of these shards and move it to the other location?

0 Answers

Read More

IdentityServer and ADFS

I'm trying to setup IdentityServer to use ADFS for authentication. The flow will be:

User -> Custom app -> IS -> ADFS

I've setup almost everything, but I'm stuck at the communication between IS and ADFS. The user seems to login successfully in ADFS, but I get an error:

ID4037: The key needed to verify the signature could not be resolved from the following security key identifier 'SecurityKeyIdentifier

when I get back to IS.

It's obvious that there's an issue with the token signing certificates in one side or the other. I've tried unsuccessfully to find some documentation explaining the relation between different certificates.

Right now I have a self signed certificate in IS that is signing tokens (set up using SigningCertificate property of IdentityServerOptions) and I have a AD certificate configured in ADFS to sign tokens.

Is there any guide or recommendation on how to properly do it? Should it be the same in both or should I configure something else to make it work?

EDIT With Fiddler I can see that everything inside ADFS runs fine and the error is when the results are posted to IdentityServer. The XML posted in wresult param is:

<t:RequestSecurityTokenResponse xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">   <t:Lifetime>     <wsu:Created xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2017-06-20T12:25:31.148Z</wsu:Created>     <wsu:Expires xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2017-06-20T13:25:31.148Z</wsu:Expires>   </t:Lifetime>   <wsp:AppliesTo xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy">     <wsa:EndpointReference xmlns:wsa="http://www.w3.org/2005/08/addressing">       <wsa:Address>urn:identityServer</wsa:Address>     </wsa:EndpointReference>   </wsp:AppliesTo>   <t:RequestedSecurityToken>     <saml:Assertion MajorVersion="1" MinorVersion="1" AssertionID="_fd1a14cd-4d18-407b-97d4-9f9dfcacd29a" Issuer="http://ssosrv.mydomain.com/adfs/services/trust" IssueInstant="2017-06-20T12:25:31.148Z" xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion">       <saml:Conditions NotBefore="2017-06-20T12:25:31.148Z" NotOnOrAfter="2017-06-20T13:25:31.148Z">         <saml:AudienceRestrictionCondition>           <saml:Audience>urn:identityServer</saml:Audience>         </saml:AudienceRestrictionCondition>       </saml:Conditions>       <saml:AttributeStatement>         <saml:Subject>           <saml:NameIdentifier>user@mydomain.com</saml:NameIdentifier>           <saml:SubjectConfirmation>             <saml:ConfirmationMethod>urn:oasis:names:tc:SAML:1.0:cm:bearer</saml:ConfirmationMethod>           </saml:SubjectConfirmation>         </saml:Subject>         <saml:Attribute AttributeName="emailaddress" AttributeNamespace="http://schemas.xmlsoap.org/ws/2005/05/identity/claims">           <saml:AttributeValue>name.surname@mydomain.tv</saml:AttributeValue>         </saml:Attribute>         <saml:Attribute AttributeName="name" AttributeNamespace="http://schemas.xmlsoap.org/ws/2005/05/identity/claims">           <saml:AttributeValue>Name Surname</saml:AttributeValue>         </saml:Attribute>         <saml:Attribute AttributeName="upn" AttributeNamespace="http://schemas.xmlsoap.org/ws/2005/05/identity/claims">           <saml:AttributeValue>user@mydomain.com</saml:AttributeValue>         </saml:Attribute>       </saml:AttributeStatement>       <saml:AuthenticationStatement AuthenticationMethod="urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport" AuthenticationInstant="2017-06-20T12:25:31.039Z">         <saml:Subject>           <saml:NameIdentifier>user@mydomain.com</saml:NameIdentifier>           <saml:SubjectConfirmation>             <saml:ConfirmationMethod>urn:oasis:names:tc:SAML:1.0:cm:bearer</saml:ConfirmationMethod>           </saml:SubjectConfirmation>         </saml:Subject>       </saml:AuthenticationStatement>       <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">         <ds:SignedInfo>           <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />           <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />           <ds:Reference URI="#_fd1a14cd-4d18-407b-97d4-9f9dfcacd29a">             <ds:Transforms>               <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />               <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />             </ds:Transforms>             <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />             <ds:DigestValue>6CeXXXXXXXXXXXXXXXXXXXX=</ds:DigestValue>           </ds:Reference>         </ds:SignedInfo>         <ds:SignatureValue>q9hJBFFFFFFFFFFFFFFFFFFFF==</ds:SignatureValue>         <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">           <X509Data>             <X509Certificate>MIIFnzXXXXXXXXXXXXXXXXXXXX</X509Certificate>           </X509Data>         </KeyInfo>       </ds:Signature>     </saml:Assertion>   </t:RequestedSecurityToken>   <t:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</t:TokenType>   <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>   <t:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</t:KeyType> </t:RequestSecurityTokenResponse> 

Thank you, Albert

2 Answers

Answers 1

Solved it. It was not related with ADFS integration, but how I had setup federation authentication in Identity Server. I was using two federation authentication identity providers: this one with ADFS and another using WinAuth. Without a callback the response from ADFS was being handled by WinAuth, so I configured different callbacks for each of them and it's working.

Answers 2

From memory:

• You need to convert the IS certificate to .cer format.
• In mmc, right click on the certificate and “All Tasks” / “Export”.
• Click through Export Wizard selecting: “No, do not export the private key”. “DER encoded binary X.509 (.CER)”.
• Select file name to export to and “Save”.
• Review options and “Finish”.
• In the ADFS wizard, import the .cer file into the Certificates tab.

Read More