It's Good To Want Things
In my previous post, My Wish List for FlexNet Manager, I detailed what I would like to see from a license analytics perspective. However, there is another product from Flexera that I use which I pay for indirectly — FlexNet Publisher.
This product showcases the oddity of Flexera’s internal business structure. Flexnet Manager is one product group and Flexnet Publisher is a completely separate product group. As far as I can tell, they are separate lines of business, with separate SKU’s and maintenance plans. And Flexera is VERY dogmatic about only providing support for people who pay maintenance for one product or the other. You can read more about FlexNet Publisher here. To boil things down to their essence:
- FNMEA is a product used to gather license usage data, generate reports and analytics about license usage and provide some very basic monitoring capabilities for FNP-based license services. This product replaced SAMSuite.
- FNP is the traditional FLEXlm license manager product. It includes source code for software producers to embed in their applications to enable licensing as well as the lmgrd binary and the base for creating a vendor daemon.
Here’s the rub. As a software consumer, Flexera will not sell me the FNP product. I’ve actually tried to purchase it. I can honestly say that I’ve never come across a salesperson in my life before or after who seemed completely uninterested in selling me something. (As a side note, they’ve also no-bid a professional services engagement, another first).
So here we are. I live and die by lmutil and lmstat and I am COMPLETELY ON MY OWN FOR SUPPORT. The Flexera folks will retort by saying that if I am having an issue with FNP that I need to work with the software supplier, say Cadence. And, in fairness, I can’t think of many times when I’ve had a problem with lmgrd or lmstat or anything else. However, I DO have enhancement requests I’d like to see (which I’ll get to in a moment). So now in order to have an enhancement request in for consideration, the “proper” way to do this is to go through my software supplier for the request, since they are in fact direct customers of the FNP product.
But *which* supplier am I supposed to use? Should I put the request in to all of them and hope that if enough of them forward it along that something may come of it? Do I work with a single one? It’s unclear. And frankly, Flexera’s track record on delivering on customer requests is shaky at best. Add another layer of indirection, and I don’t hold out a lot of hope.
I am therefore summoning the Powers of the Internet and making a public declaration of things I’d like to see in FNP.
I don’t know about you, but we’ve got a process that’s constantly doing an lmstat and putting the usage data into a database to track concurrent utilization. We’ve even got things that parse out who is using how many and then linking that user data to what site they are at or what LSF cluster they are using. A couple of years ago, we upgraded the “default” version of lmstat, and lo and behold, the output format changed. So we had to spin up a project to test against the new version and have an orderly transition. It would have been so much better if we had an invariant format for this output.
FNMEA also keeps track of concurrent usage, but I’ve always found it limiting. Particularly when you consider the use case of marrying username data with things like business unit, site, etc.. FNMEA is more geared as an OLAP interface, but my use case is closer to OLTP model. In the spirit of my other wish list, just let me get my data and put it in the systems I already have investments in and where I can combine it with other, non-license data so that I can get a complete picture of my environment. FNMEA will never be able to do that.
A version-invariant output from lmstat would make this easier and more resilient.
To The Microsecond Resolution In lmstat And Report Logs
For any of you who have had to process logs from a compiler that uses Flex, you will understand what I’m talking about here. Compilers are very chatty with Flex and there are TONS of transactions that have the checkout and checkin time as exactly the same. These “zero second checkouts” are problematic for the algorithms used to create after-the-fact concurrent usage graphs. Using epoch time to timestamp transactions in the report logs has served us well for 30 years, but it’s time we got more granularity.
The other side of this is lmstat output. If I’m using lmstat to populate a concurrent usage database, today my best granularity is by the minute. It’s a small thing to add seconds to that, and it would be useful information for me. Why? Because every bit of information when you are dealing with multi million dollar software contracts is useful. Also, it’s fundamentally my data and I’d like to consume it the way I’d like to consume it.
LM_PROJECT in debug logs and lmstat output
Combining license and job data together is a goal most HPC shops have. But doing so requires that you have a common key between the two so that you can stitch together the millions of transactions reliably. And while there is value in providing OLAP-style reports on past usage, there is also value in being able to make realtime decisions on job scheduling. Let’s say, for instance, that I have the most important project in the company and it’s GOT to get out. I might prioritize work for a period based on their project tag. My job scheduler lets me see the project tag on a job, and if I had project tags on licenses I could then tie scheduling decisions based on license availability and on project tag. Or perhaps we want to see a realtime snapshot of which projects are using which licenses in order to debug a throughput problem. Today, the LM_PROJECT variable is buried in the report log, which I have to rotate, import and then potentially export. (Kudos by the way to Synopsys for giving me the ability to at least spit out LM_PROJECT into the debug log through the use of their advanced logging. That’s a very useful feature.)
To echo a theme, in the end this is my data and I’d love to access it in ways that are convenient to me for whatever purposes I deem. In every other aspect of industry, data is becoming more available and more realtime. It’s time Flexera followed suit and let me get LM_PROJECT in real time for some reasons I can enumerate, but for also for reasons I haven’t yet conceived.
True High Availability
A colleague of mine, Joshua Hauta, among his many pithy sayings, likes to say that we want to see “nines as far as the eye can see” when it comes to license server availability. He couldn’t be more right. At the end of the day, none of the work you do on analytics matters if people can’t count on your license servers to be up and responsive. Flexera’s answer to high availability? License Triads. Now, I know there are those of you who use triads and even like triads. I am not in that camp. We will have to respectfully disagree on this point, and that’s OK.
If you search for FLEXlm license triad, you will come upon this blog post from Flexera (dated 2013). If you take a moment to read this, you will see that Flexera themselves are advocating not to use triads. Rather, Jim Griffin suggests that you simply distribute your licenses amongst three separate, independent servers. I have no quarrel with that suggestion and make use of it myself as a normal course of business. However, splitting your pool into multiple servers introduces new problems. What if you want a MAX restriction on a user or group? Now you have to decide whether they should have that limit on a single server and not get any licenses on the other two. Or perhaps you give them a max of 1/3 of their total on each server. Neither solution is efficient, nor satisfying.
In more modern architectures, you would query the license server via a REST type interface over HTTP/S. And individual servers would be disposable and on an autoscale group, hidden behind a load balancer.
But FLEXlm is a 30-year old technology, based on stateful TCP connections. Now, some enterprising Flexera salesperson is right now licking their chops, waiting to tell us more about FlexNet Embedded, their next generation license technology. I don’t claim to know a lot about this, but what I have heard leads me to believe that it adheres to a more modern architecture.
And herein lies the conundrum. I call it Flexera’s “Itanium Moment”. Flexera has developed a new architecture that is more suited to the modern world. But here’s the rub. This new technology is completely incompatible with their established product. However, they now have a legacy of almost two decades of displaying outright contempt for their customers. They are famous for not listening. I have heard them start flat out refusals to do things with “Quite frankly…”. They no longer release documentation to end users like myself for lmgrd or lmutils because of a “legal issue”. Apparently, we’re supposed to get that from suppliers like Cadence or Synopsys.
So the question is, if we have to change, why should we choose them? And it’s a fair question. They’ve behaved like a monopolist. Why? Because they effectively are in the Semiconductor industry. In business school, you learn about the BCG Matrix, named for the famous Boston Consulting Group. The matrix forms the famous 2×2, graphing industry growth against market share. To cut to the chase, those who have high market share in a low growth industry need to foster their Cash Cow. And you do so by slashing R&D and doing the bare minimum in order to maintain that high market share. At the same time, you charge the highest cost you can get away with to derive the maximum economic benefit. And that is precisely what Flexera has done with FlexNet.
I’ve drifted a bit from my primary point, which is this. I’ve got legacy FlexNet licenses which I will have for a long time, FlexNet Embedded or not. We have a significant economic investment in these licenses, and I must keep them up 99.99999% of the time, if not more. Give us a supportable way to use tools like Veritas VCS or integrate nicely with VMotion for VMWare VM’s. There are people who have gone to great lengths to engineer solutions around the technology to achieve their goals without using Flex triads. I’ve seen them in action, and they are fantastic. But at their hearts, they are hacks. Flexera needs to evolve their existing product and help ease the transition into the new wave of technology or they will face a hoard of customers who will take their perpetual licenses and say adieu.
API Accessibility (License Reservation)
My company makes use of IBM/Spectrum License Scheduler. The tool does a fair job of allocating scarce license resources (in cluster mode), but let’s face it. This solution is a hack. It’s a hack because it allocates “tokens” which are meant to represent an actual license. It then schedules jobs based on this token representation of a real license, which may in fact get yanked out from underneath it before the job actually gets a chance to check out the license.
Why? Because there is no programatic API with which to interact with Flex. If LSF could actually communicate with the license server to reserve a license, then the policy would have teeth. But, it doesn’t. An effort was made several years ago to add this functionality. And one major EDA vendor actually was on the cusp of releasing it. And then… something happened. The project got tabled, and we’ve all been left with trying to manage our resources using the same technology we’ve fundamentally had for 30 years.
Give us a REST API. Let us be able to do lmstats, giving JSON or CSV or XML output. Let us reserve licenses in advance by passing a token along that the job uses to validate when it actually needs the license. Let us automate. For that is the compulsion of the age — automation. We can only go so far with what we have now.
Leave a Reply