Whenever someone accesses the company network via the Wi-Fi at work, a workstation, or through a VPN, their computer is assigned a temporary IP address through DHCP. If these devices are discovered with an out-of-the-box Universal Discovery configuration, UCMDB reconciliation rules often end up relating just a few IP address CIs to hundreds or thousands of Computer CIs. This is essentially akin to thousands of people sharing a single telephone number. Unsurprisingly, this kind of problem can quickly get out of hand, leading to all sorts of confusion and inaccuracies: exponential mis-merging of CIs, constant flip-flopping of attributes (between up-to-date and out-of-date or vice versa), and generally poor performance caused by misidentification and confusing reconciliation outcomes.
In this post we’ll take a little time to explain the problem and to offer a solution.
The Issue and Investigation
In general terms, legacy HPE DDMi has tended to focus on deployable assets (such as personal computers, printers and non-data center/managed servers) and it did this job very well. DDMA users, meanwhile, have historically eschewed discovery scope such as desktops because of perceived performance limitations, instead focusing on application, database, and server discovery for relationships. With the merging of the DDMI technologies into the DDMA & UCMDB product, it was only a matter of time before strange and remarkable inconsistencies between the two approaches occurred.
For instance, we were recently helping an HPE Universal Discovery customer who was attempting to use both DDMI and DDMA functionality. This user reported an issue to us wherein temporary IP addresses (IP Addresses with the IP Lease Time attribute set to “short”) from DHCP clients with VPN connections were merging and reconciling in an illogical way. The example we saw in the customer environment was a single IP address that was related to over 1800+ unique Computer CIs and 1700+ unique Interface CIs. Normally, Computers do not share non-virtual IP addresses, so seeing a single IP Address CI related to that many Computers and Interfaces warranted some investigation.
Looking more closely at the Computer results didn’t reveal anything extraordinary. But once we started looking at the Interfaces we found some correlating evidence. Each Interface in question was a virtual adapter created in Windows for use over a VPN. They only had one thing in common: each Interface had the same value for Mac Address. This fact will play a role later, but first we had to continue gathering evidence by looking at the IP Address CIs themselves.
Looking more closely at the IP Address CIs, we noted pervasive attribute flip-flopping in the CIs’ histories. In particular, we noticed that the IP Address attribute had many different IP Addresses over time, and was related at many points in time to many other Interface CIs. By looking into the Discovery jobs that modified the CIs, we were able to see that nearly all attributes (even the IPs!) were being constantly changed back and forth by the Host Connection by Shell and Inventory Scanner Discovery jobs.
But the order and pattern was illogical. While attributes that shouldn’t be changing were, the following attributes remained constant: IP Address Lease Time (short), Routing Domain (the company domain), and IP Address Type (IPv4). Of course, these three were only remaining constant because every IP Address CI in the entire customer environment shared the same value for each one of the three attributes (and when UCMDB doesn’t detect a delta, it doesn’t perform an update). It just so happens that these attributes are also key identification attributes for the IP Address CI type in the OOTB reconciliation rules. So it appeared that we were dealing with a problem in reconciliation.
We realized that the DDMI scan files contain critical information about the Computer, Interface, and IP Address. But this information is often factually older than the DDMA Networking results. So the flip-flopping pattern during reconciliation was largely caused not by what data was correct, but rather by when it was correct, since scan files can contain snapshots that are weeks old. This exacerbated the problems of changing DHCP IP Addresses, which Interface would be related to which IP Address, and (when these devices were connected to VPN or wireless) even to which Node.
A single instance of this reconciliation problem would be confusing in and of itself. But when compounded over time, the results are badly damaged CI quality at the Computer level, and obvious problems with relationships to IP Addresses, Interfaces, and Layer 2 data. Not only is data quality badly impacted, but performance costs and wasted overhead in UCMDB History, reconciliation processing, and Discovery are severe.
Out-of-the-box reconciliation rules fail when non-server devices are discovered because they do not appropriately handle an environment that uses short IP address lease times for DHCP or VPN clients on devices such as desktops or kiosks.
The OOTB reconciliation rule for IP Address with Short IP lease time uses Routing Domain and the MAC Address of the connected Interface CI, or ARP MAC. ARP MAC is a hidden attribute, but most environments will have IP Addresses with blank ARP MACs or ARP MACs marked as “NA”.
However, many environments use virtual interfaces for VPN connections which don’t use hard-coded or unique MAC addressing schemes (this is a problem with virtual Interfaces and VMware as well). Meaning thousands of interfaces are likely to share an identical MAC Address. And since there may be thousands of temporary IP Addresses that have identical Routing Domains and MAC Addresses for the connected Interfaces. Reconciliation will then see these matching conditions and then attempt to apply a validation rule to determine what action to take.
In order to validate these similarly identified temporary IP addresses, the first reconciliation priority validation rule method employs IP address type (IPv4) which, as we stated earlier, are identical values for IPs used by DHCP and VPN. The second priority (employed when the first rule set fails to determine uniqueness) to validate two similarly identified IP addresses are Routing Domain and the actual IP address Name (e.g. 10.0.0.1). In other words, the sole attribute (besides IP address value) that could prevent a data conflict for a given IP address takes second priority in validating uniqueness!
Get it? Interfaces and IP Addresses all match per the ID rules, so IP Addresses are approved by the validation process to merge. Then reconciliation can’t undo the mess it is forced into by the Identification Rule. Fun stuff, right?
So in order to prevent this scenario from impacting Nodes and more complex relationships such as Services, Layer 2 and Application maps, you need to modify several CI Type reconciliation rules. We provide a very reliable solution below.
Note: Click here for a zip file with the relevant XMLs. We have included both the suggested reconciliation rules and the OOTB reconciliation rules for reference comparison.
As a solution to the issue of IP addresses with short IP lease times from DHCP and VPN improperly merging and reconciling to one another, we suggest the following:
1. Delete the containment relationships between nodes and interfaces with problematic IP addresses with short IP lease times (you can use an enrichment for this).
2. Change the reconciliation rule for IP addresses with short IP lease times (any ip_lease_time with the value of “1”) to include the related Node’s name, then move the validation priority of the IP Address’ name and routing domain for short IP lease time IPs up to Priority 1.
See attached XMLs for recommended reconciliation rule
3. Change the reconciliation rule for Node. OOTB reconciliation rule uses only 66% of the MAC address of the related Interface, which is not a unique value and commonly represents just the manufacturer and model of the interface. (This is silly, since only the last few digits would denote any difference.) That’s why this customer ended up with over 1700 interfaces with identical MAC addresses. We make this recommendation constantly for many different reconciliation use cases.
We recommend changing the reconciliation rule from 66% of the IP address, IP address’ authoritative DNS name and related interface’s Mac Address to 100%. The attached recommended reconciliation rule for Node changes the 66% match condition to 100%. It also adds the interface’s index and interface speed as further identifiers.
4. The OOTB adapters for Host Connection by Shell, WMI, Powershell, and SNMP have auto-delete for Containment relationship set to “on” by default. When a system connects via DHCP or VPN and is assigned a new IP address, the Host Connection discovery job should delete the previous containment relationship to previous IP addresses used. Run the appropriate Host Connection discovery job(s) for your environment and verify that the relationship between systems and their old DHCP/VPN IP addresses are being deleted.
5. Now run your discovery jobs as usual. Range IPs by ICMP, Host Connection by Shell, and the scanner jobs. Validate your results. You may also wish to advance the candidate for deletion and actual deletion period settings for Interface and IP Address to less than 60 days.
We strongly recommend that you implement these changes:
- Regardless of whether you use DDMA and DMMI together.
- Regardless of whether you discover desktops, laptops, or wirelessly connected devices yet.
- Because it makes sense to let reconciliation operate correctly, free of unnecessary ambiguity.
- Because you should be able to discover all your servers and devices.
- Because The Internet of Things is coming and you should be ready for it. We’ve already shared solutions with you to fix the performance and scalability issues that would have impacted your UCMDB enough to limit your Discovery use case.
If you have questions or would like support to implement these changes in your environment, please attend our twice monthly free support sessions, held every other Wednesday at 8AM, Pacific Time (refer to sidebar for next session) or contact Effectual directly by clicking below